gatsby-source-filesystem: create-remote-file-node does not handle retries

[schoeneu]

Description

In https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L162 , gatsby-source-filesystem tries to set up retries in case the remote connection breaks when creating a new file node.

However, the "got" documentation states that for "got.stream", the retry-option is ignored:
https://github.com/sindresorhus/got/blob/master/readme.md#retry

Note: When using streams, this option is ignored. If the connection is reset when downloading, you need to catch the error and clear the file you were writing into to prevent duplicated content.

Steps to reproduce

Use a plugin that uses create-remote-file-node (for example remote-images) with a flaky server giving 502s sometimes. No retries will happen.

Expected result

5xx errors from server lead to retries.

Actual result

Instant breakage.

[vladar]

Thank you for opening this, @schoeneu

Gatsby uses got@^8.3.2 branch which doesn't have this quote in docs. But you are probably right and even if version 8 of got supports retries for streams - they are probably broken anyway. Here is a quote from one of the got issues about this (https://github.com/sindresorhus/got/issues/871#issuecomment-530496244):

This is because the connection may get broken while downloading, and if the retry functionality was here you would get duplicated content. To "retry" properly you need to make your own function that will handle the errors - then make new stream and pipe it again (if you were writing to a file, don't forget to clear it).

The point is that Got cannot clear the file you were writing into.

Also related: https://github.com/sindresorhus/got/issues/309

Which makes sense. So we do need to account for this in gatsby-source-filesystem. PR to address this is more than welcome! 💜

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.
Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜

[axe312ger]

This PR would benefit from a fix on this as well: https://github.com/gatsbyjs/gatsby/pull/24288

It would also allow https://github.com/gatsbyjs/gatsby/pull/15783 to use createRemoteFile again :)

[axe312ger]

@pieh I think this ticket was about retry not working. The PR now removed the (broken) retry.

Still, if we get a network error, create-remote-file-node will still break the whole build instead of retrying :(

IMHO this is not fixed

[pieh]

Oh, I didn't notice the PR was linked to the issue, re-opening as functionally nothing changed

[axe312ger]

@pieh ohh 👌