Gatsby: [gatsby-source-filesystem] why createRemoteFileNode failing still resolves the promise?

[antoinerousseau]

Summary

When createRemoteFileNode fails to download the remote file (e.g. 404), it still resolves with an empty fileNode, and writes Failed to process remote content {url} to the console:

https://github.com/gatsbyjs/gatsby/blob/a358239be945485388601159e7f18cf4a5c17cc8/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L126
https://github.com/gatsbyjs/gatsby/blob/a358239be945485388601159e7f18cf4a5c17cc8/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L285-L287

Why not just reject(error) instead of console.warn() & resolve()?
That would make more sense allowing us to handle the error as we wish, and would avoid polluting the terminal.

Relevant information

Environment (if relevant)

  System:
    OS: macOS 10.14.3
    CPU: (8) x64 Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz
    Shell: 5.3 - /bin/zsh
  Binaries:
    Node: 11.6.0 - /usr/local/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.8.0 - /usr/local/bin/npm
  Languages:
    Python: 2.7.15 - /usr/local/bin/python
  Browsers:
    Chrome: 72.0.3626.119
    Firefox: 65.0.1
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.20 => 2.1.20
    gatsby-image: ^2.0.31 => 2.0.31
    gatsby-plugin-manifest: ^2.0.21 => 2.0.21
    gatsby-plugin-netlify: ^2.0.11 => 2.0.11
    gatsby-plugin-offline: ^2.0.24 => 2.0.24
    gatsby-plugin-react-helmet: ^3.0.7 => 3.0.7
    gatsby-plugin-sharp: ^2.0.24 => 2.0.24
    gatsby-plugin-styled-components: ^3.0.6 => 3.0.6
    gatsby-plugin-typescript: ^2.0.10 => 2.0.10
    gatsby-source-filesystem: ^2.0.23 => 2.0.23
    gatsby-transformer-sharp: ^2.1.15 => 2.1.15
  npmGlobalPackages:
    gatsby-cli: 2.4.12

File contents (if changed)

gatsby-config.js: N/A
package.json: N/A
gatsby-node.js: N/A
gatsby-browser.js: N/A
gatsby-ssr.js: N/A

[DSchau]

This seems like a reasonable change. Would you like to try your hand at making this change?

Will want to validate that rejecting doesn't entirely fail the build (which we probably wouldn't want?) and provides actionable steps to fix.

[antoinerousseau]

@DSchau done @ #12348 :)

@tsimons any insight on why you initially returned resolved Promises instead of rejected ones?

[antoinerousseau]

Also, what about disabling that progress bar? It fills the terminal when downloading many remote files (and CIs don't like that). The progress feedback could be made optional, defaulting to true.

Edit: ok it's not that bad.

[DSchau]

@antoinerousseau we'll want to keep the progress bar :)

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[antoinerousseau]

Not stale! (can't add labels)

[kepi0809]

any news according to this? about 1/4 my teaser-imgs are failed to process :/

[bakeruk]

Check my answer in #12848

[leonfs]

I added some comments to that opened PR. I don't think that fix is a breaking change.

If worried about cascading the issues to consumers, is probably good that they see that some of their remote files' download fails, they might not have noticed it and have broken sites (it happened to us and we caught it because of synthetic tests), especially if the number of pages is high.

[bakeruk]

Out of curiosity, try updating to [email protected] running yarn upgrade and let me know if your issues are fixed.