Gatsby: createRemoteFileNode can't deal with special characters

[gustavpursche]

Description

In some cases localFile was null, when pulling media from wordpress.

Steps to reproduce

I ran into the issue, while working with gatsby-source-wordpress:

export const fragment = graphql`
  fragment imageTextCombination on WordPressAcf_imageTextCombination {
    image {
      localFile {
        childImageSharp {
          fluid(maxWidth: 500) {
            src
            srcSet
          }
        }
      }
    }
  }
`;

In some cases image.localFile was null which made me wonder. After some time I found out, that the difference in the files appeared to be an ä (a german Umlaut), which is currently not normalized by default in wordpress.

Installing a plugin to normalize fileNames on upload fixed the issue.

I think the root of the issues comes from here:

https://github.com/gatsbyjs/gatsby/blob/1257a225de0bb3be8314ecc5684d1c6635d3ee05/packages/gatsby-source-filesystem/src/create-remote-file-node.js#L344-L348

I've checked valid-url and added a test:

t.ok(is_uri('http://localhost/ä'), 'http://localhost/ä');

which would fail.

Expected result

image.localFile should contain the file and not care about the fileName at all. I'm not sure whether I'm right on this, that's why I opened an issue instead of an PR. What do you think?

Actual result

image.localFile is null.

Environment

  System:
    OS: macOS 10.14
    CPU: x64 Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz
    Shell: 5.7.1 - /usr/local/bin/zsh
  Binaries:
    Node: 11.11.0 - /usr/local/bin/node
    Yarn: yarn install v0.21.3
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
Done in 47.06s. - /usr/local/bin/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  Browsers:
    Chrome: 69.0.3497.100
    Firefox: 62.0.3
    Safari: 12.0
  npmPackages:
    gatsby: ^2.2.1 => 2.2.1
    gatsby-image: ^2.0.34 => 2.0.34
    gatsby-link: ^2.0.16 => 2.0.16
    gatsby-plugin-react-helmet: ^3.0.10 => 3.0.10
    gatsby-plugin-react-svg: ^2.1.1 => 2.1.1
    gatsby-plugin-sharp: ^2.0.29 => 2.0.29
    gatsby-plugin-styled-jsx: ^3.0.5 => 3.0.5
    gatsby-source-filesystem: ^2.0.27 => 2.0.27
    gatsby-source-wordpress: ^3.0.47 => 3.0.47
    gatsby-transformer-sharp: ^2.1.17 => 2.1.17
  npmGlobalPackages:
    gatsby-cli: 1.1.58

[coreyward]

@tsimons wrote this code, perhaps he can shed some light.

[sidharthachatterjee]

Thank you for pointing this out @gustavpursche

Just took a look at the code for the valid-url package that we use. The regular expression here seems to be the issue
https://github.com/ogt/valid-url/blob/8d1fc52b21ceab99b68f415838035859b7237949/index.js#L62

A great next step would be to open an issue in that repository and maybe a failing test. We can then hear what the author has to say and get it resolved. Would you like to an open an issue in https://github.com/ogt/valid-url/issues?

[gustavpursche]

@sidharthachatterjee I think you are right. After reading a bit about the difference of IRI and URI today, I also think the regex shouldn't be part of the authors is_iri function.

However - the author hasn't replied to any issues since 2015 or commited anything since then. Similar issues are open already, so maybe the only thing that could convince him is, that Gatsby is so popular. I'll try :)

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

Thanks for being a part of the Gatsby community! 💪💜

[gustavpursche]

@gatsbyjs-read-only-user Not stale.

[gustavpursche]

Hi there again. Thanks to the bot reminder: the author of the lib didn't respond (since 2015). I think the whole module could be small enough to inline the code. What do you think?

The other solution could be to fork the project and rework the code based on the actual specs and to publish it as a new npm module.

[domvo]

@gustavpursche could you tell me what plugin you used to normalize the filenames? Thanks a lot!

[gustavpursche]

@domemvs Sure! It's called "Clean Image Filenames".

[domvo]

Thank you, that was quick!

[daniloef]

Hi! Do you know a plugin that "clean" the filenames of uploaded images?

[KyleAMathews]

https://www.npmjs.com/package/sanitize-filename looks like a good one

[gabrielloeb]

Begging for mercy since I don't understand the fullness of what's happening inside gatsby-source-filesystem, but can the change made in #15835 be applied directly to that plugin? I'm experiencing the same issue with gatsby-source-prismic, and I presume other source plugins are falling (or will fall) victim to this issue. Is there any reason not to attempt to sanitize urls before abandoning them with a message to the console?

wrong url: https://images.prismic.io/redacted/b56f83c2-73e9-4be7-bb53-0792209c74df_CarréTopHandleMini_.jpg?auto=compress,format

[yann-yinn]

I had the same issues with filenames from a S3 bucket with accented characters:

example of a problematic filename téléphonefixe.jpg
error message :

here was an error in your GraphQL query:

Unexpected error value: "url passed to createRemoteFileNode is either missing or not a proper web uri: (...)-téléphonefixe.jpg" 

I tried to use encodeURI

          return createRemoteFileNode({
            url: encodeURI(source.image.url),
            store,
            cache,
            createNode,
            createNodeId,
            reporter
          });

but then got another error because resulting filename seems to be badly encoded (double encoded ?):

t%25C3%25A9l%25C3%25A9phonefixe.jpg

It supposed to be t%C3%A9l%C3%A9phonefixe.jpg ...

PS: @daniloef i'm using a combination of https://www.npmjs.com/package/sanitize-filename and https://www.npmjs.com/package/slug to create clean filenames on upload