Gatsby: [gatsby-source-filesystem] Broken Basic Auth on createRemoteFileNode

[patricknick]

Description

Basic auth headers seem not to be set correctly anymore in the latest version. Instead in the request header only a "auth":"some_user:some_pass" are set without proper encoding.

The broken change seems to be introduced here: https://github.com/gatsbyjs/gatsby/commit/a358239be945485388601159e7f18cf4a5c17cc8

Steps to reproduce

It's a bit hard to test, I had to intercept my requests to the API to figure out what's happening... Maybe use Charles?

Code in my use case (like described here):

const auth = {
htaccess_user: "some_user",
htaccess_pass: "some_pass"
}

const fileNode = await createRemoteFileNode({ url: pictureURL, store, cache, createNode, createNodeId, auth });

Expected result

I expect a header like this:
"Authorization":"Basic aW5mb0Brb25vbasfwefwfsadaweawefweEZbTn03RkUrVlJFZWZ0bw=="

Actual result

"auth":"some_user:some_pass"

Environment

System:
OS: macOS 10.14.3
CPU: (8) x64 Intel(R) Core(TM) i7-6920HQ CPU @ 2.90GHz
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 10.12.0 - ~/.nvm/versions/node/v10.12.0/bin/node
Yarn: 1.13.0 - /usr/local/bin/yarn
npm: 6.8.0 - ~/Development/tlc-website/plugins/gatsby-source-tocco/node_modules/.bin/npm
Languages:
Python: 2.7.10 - /usr/bin/python
Browsers:
Chrome: 72.0.3626.119
Firefox: 65.0.1
Safari: 12.0.3
npmPackages:
gatsby-node-helpers: ^0.3.0 => 0.3.0
gatsby-source-filesystem: 2.0.23 => 2.0.23
npmGlobalPackages:
gatsby-cli: 2.4.12

[DSchau]

Sorry for the late reply on this!

Would you be able to provide a reproduction for this?

It will help us validate the regression _and_ also demonstrate the issue. If there are any authentication details necessary, feel free to DM me on Twitter or share them however you prefer!

Merely seeing the failure would help us trace this down, though, so let's start there!

It does seem like the change has introduced more correct behavior, so interested to see what went awry here!

[patricknick]

I created a dummy project:
https://github.com/patricknick/gatsby-source-filesystem-bug-repro

It's a very basic and simple setup but probably the easiest for you to reproduce. It points to the service https://httpbin.org/headers, which returns the request headers in the body.

How to debug:

1. Download the repo and run yarn to install the dependencies
2. Run yarn start
3. Inspect the .cache folder --> .cache/gatsby-source-filesystem/--some-random-guid--. In there you should see a file with the response from the httpbin service.

Testcases

1. The repo currently installs gatsby-source-filesystem version 2.0.22. It returns something like this:

{
  "headers": {
    "Accept-Encoding": "gzip,deflate", 
    "Authorization": "Basic VVNFUjpQQVNTV09SRA==", 
    "Host": "httpbin.org", 
    "User-Agent": "got/7.1.0 (https://github.com/sindresorhus/got)"
  }
}

1. Install the latest version of gatsby-source-filesystem ^2.0.23. It returns something like this:

{
  "headers": {
    "Accept-Encoding": "gzip,deflate", 
    "Auth": "USER:PASSWORD", 
    "Host": "httpbin.org", 
    "User-Agent": "got/7.1.0 (https://github.com/sindresorhus/got)"
  }
}

[skinandbones]

Yup, I can see that now. Previously, headers was being used to send a set of options (including auth) down to the got http client. That was fixed in a358239 and, I agree, is the cause of the regression since it has exposed this other issue of mixing up headers and http client options into one thing. We'll need to sort that out.

[skinandbones]

I think the approach to take on this one should be to extend requestRemoteNode to accept a http_options argument and then pass the auth options via the new argument. In requestRemoteNode, reverse merge the default http timeout and retries options into http_options before handing off to got.