Gatsby: [gatsby-plugin-sharp] Regression of the thumbnail generation bug

[iAdramelk]

The bug described in https://github.com/gatsbyjs/gatsby/issues/19576 and fixed in 2.3.3 returned in 2.4.0.

After upgrading plugin to 2.4.0 it started generating thousands of thumbnails and regenerating them after each change, downgrading plugin to 2.3.13 fixed bug.

[sidharthachatterjee]

@iAdramelk Do you happen to have a minimal reproduction for this? That should allow us to take a look and resolve this as soon as possible.

[wardpeet]

Hey @iAdramelk,

Can you check your build times as well? The latest upgrade seems like a regression but is actually not. We now log all transform operations even if they won't be processed. So it might say 300k images but actually only run 1000 images.

Previously the cache was inside gatsby-plugin-sharp but it's now moved to gatsby itself to make sure we can restart images processing when gatsby crashes and open the door to lazy image resizing on develop.

[iAdramelk]

@wardpeet build time slightly increased, but not very much. The main problem is not static build time, but that in development mode, after each update, all images are regenerated. Basically: Add 1 symbol to md file, save, and wait for 30-60sec for all images to rebuild.

@sidharthachatterjee Sorry don't have it ATM, can create it tomorrow if examples from old tickets is not enough.

[iAdramelk]

@bartol 2.3.13 is ok.

[pieh]

Note here after some investigation:
I tried reproducing this with gatsby-starter-blog and editing markdown file with some images ( https://github.com/gatsbyjs/gatsby/blob/master/starters/blog/content/blog/hello-world/index.md ):

We do get progress bar, but image generation doesn't actually happen (we reuse previously generated ones). Only way I could trigger true image processing is when I actually add new image to that markdown file.

I do see some extraneous query running, so maybe this is more related to long wait times?

In any case - the simple reproduction I used doesn't actually re-create extra images - the visual bug of showing progress bar do need to be handled, but it doesn't seem to cause long rebuild/refresh times.

Sorry don't have it ATM, can create it tomorrow if examples from old tickets is not enough.

All reproduction from old ticket were private and I don't have access to them, so if possible to create one - that would be greatly appreciated

[iAdramelk]

@pieh sorry, can't do minimal reproduction yet, but our project is in open github repository so you can check it there.

Project: https://github.com/iterative/blog/

Commit with that had the bug: https://github.com/iterative/blog/commit/f90b2fc8e2a7749eed1c3d390481ba3b761463bd
Commit that downgraded it and fixed the bug: https://github.com/iterative/blog/commit/a773f5fc337113de3b4f3a47fbba166667aa3ab8

If it's too big I can try to make smaller example later.

To reproduce checkout commit, run yarn server, edit any of md files and then check console and image regeneration info in it.

[pieh]

That's good! I think reproduction being larger site is what contributes to the general slowness.

I do suspect it's not related to image generation as that's just visual bug (at least in my testing of very small blog), but rather to extraneous query running I mentioned in my previous comment, but will keep you posted once I check site from your repo (to see if there is image generation happening there or not)

[pieh]

@iAdramelk Can you try updating:

• gatsby package to at least 2.19.4 (there was fix related to extraneous query running)
• and gatsby-plugin-sharp to use progress-fix dist tag:

yarn add gatsby gatsby-plugin-sharp@progress-fix

this seems to fix the issues here, but want to double check

The gatsby-plugin-sharp changes have in progress / draft PR that you can track in https://github.com/gatsbyjs/gatsby/pull/20839

[iAdramelk]

@pieh sorry for the long answer. If I understand corectly, fix was alredy merged to master and released?

I tried to compare my current version of dependencies with the latest ones. Current one:

• "gatsby": "^2.19.1"
• "gatsby-plugin-sharp": "2.3.13"

I deleted public folder and run yarn gatsby develop and got this:

success Generating image thumbnails - 46.200s - 426/426 9.22/s
````

Then I updated to the latest versions:

- "gatsby": "^2.19.6"
- "gatsby-plugin-sharp": "2.4.2"

Deleted `public` folder again and got this:

[============================] 132.839 s 788/788 100% Generating image thumbnails
```

As you can see, number of thumbnails changed from 426 to 788 and initial development build time changed from 46s to 133s. It's to slow for me to update, so I have to stay on the old version for now.

But the bug with the thumbnails regeneration after markdown update is indeed fixed.

Both examples were tested using this commit: https://github.com/iterative/blog/commit/f2db4856fdedc83710796d51cca577bd215d30d5 Only difference is package versions.

[pieh]

I will iterate on this more

[pieh]

Ok, I tried with [email protected] and newer ones

When starting with clean cache (that means both .cache and public), the 2.3.3 one says it generates 788 as well for me which is similar to newer plugin - this one says 791 (so 3 difference).

The kicker is when I added more debug logging the 2.3.3 still actually make 791 operations (and not 788) - so there was some visual bug happening there it seems

[pieh]

So I'm inclined to say this is result of changing how hashes are generated and not deleting public directory as well as .cache between reruns

[iAdramelk]

@pieh Thanks for the update! Tried again after deleting .cache folder this time.

Before update:

success Generating image thumbnails - 96.004s - 788/788 8.21/s

After update:

[============================]   133.869 s 788/788 100% Generating image thumbnails

Looks like difference in amount of images was indeed because of the cache folder, sorry for confusion, I was wrong here.

But there is still an increase of compilation time for almost 50% (or 40s in my case). Is it possible to reduce it back?

[pieh]

We will be working on improving performance more, I do see increase in total time builds. Please use total build time here, because we were shuffling things around and the time for Generating image thumbnails step isn't 1:1 comparable with before. Tho I did notice increased total build time with newer plugin version so will keep working on that and see what exactly caused this.

[lucasnantonio]

Not sure my problem is related, but here it goes since I couldn't find anything closer to what I'm seeing.

When I edit an image, while in development mode, the terminal shows changed file... and success Generating image thumbnails.
But the image generated does not get updated to the new one.

I'm sure at some point while developing with gatsby images would update instantly in the browser while I developed. Now I need to clean cache or change the image name every time to see the change?

Here's my config

 {
      resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [
          `gatsby-remark-unwrap-images`,
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 1800,
              linkImagesToOriginal: false,
              showCaptions: true,
            },
          },
          `gatsby-remark-smartypants`,
        ],
      },
    },

[madalynrose]

hi there, @lucasnantonio! Could you open a new issue for this? It'll help us triage better and you can always link it in here if you find it is indeed related. Additionally, could you provide a minimal reproduction of your problem either by linking to your repo or by creating a simplified version in something like codesandbox? Thank you!

[FraserThompson]

Just a note that the version of Sharp used by the last version without this bug (2.3.13) requires Python 2.7 which is end of life, so it's not a great workaround.

For my very image heavy site newer versions of the plugin make the site impossible to develop on as it regenerates over 100,000 thumbnails on each build, rather than just generating what it needs to.

[hvitis]

+1 I am having the same problems

1:37:40 AM: success Generating image thumbnails - 807.589s - 1186/1186 1.47/s
1:37:40 AM: Generated public/sw.js, which will precache 20 files, totaling 749281 bytes.
1:37:40 AM: success onPostBuild - 0.174s
1:37:40 AM: info Done building in 820.666123289 sec

over 1200 images on blog with 20 posts (around 40 images)

"gatsby": "^2.4.1",
    "gatsby-cli": "^2.11.3",
    "gatsby-image": "^2.5.0",
    "gatsby-plugin-feed": "^2.1.0",
    "gatsby-plugin-google-analytics": "^2.0.18",
    "gatsby-plugin-lodash": "^3.0.5",
    "gatsby-plugin-mailchimp": "^5.1.0",
    "gatsby-plugin-manifest": "^2.0.28",
    "gatsby-plugin-offline": "^2.0.25",
    "gatsby-plugin-prefetch-google-fonts": "^1.4.2",
    "gatsby-plugin-react-helmet": "^3.0.12",
    "gatsby-plugin-sharp": "2.8.0",
    "gatsby-plugin-styled-components": "^3.0.7",
    "gatsby-plugin-typescript": "^2.0.12",
    "gatsby-remark-copy-linked-files": "^2.0.11",
    "gatsby-remark-images": "^2.0.6",
    "gatsby-remark-katex": "^3.0.4",
    "gatsby-remark-mermaid": "^1.0.0",
    "gatsby-remark-normalize-paths": "^1.0.0",
    "gatsby-remark-prismjs": "^3.2.8",
    "gatsby-remark-responsive-iframe": "^2.1.1",
    "gatsby-remark-smartypants": "^2.0.9",
    "gatsby-source-filesystem": "^2.0.29",
    "gatsby-transformer-remark": "^2.3.8",
    "gatsby-transformer-sharp": "^2.1.17",