Gatsby-source-wordpress-experimental: "gatsby develop" gets hung up on "Downloading remote files"

Created on 13 Oct 2020 · 30Comments · Source: gatsbyjs/gatsby-source-wordpress-experimental

I am trying to develop my Gatsby/Wordpress project locally but it keeps getting hung up on "Downloading remote files:"

 -> wordpress__CATEGORY fetched : 86
 -> wordpress__TAG fetched : 93
 -> wordpress__wp_topic fetched : 8
 -> wordpress__wp_users fetched : 20

Path: /wp-json/wp/v2/users/me?per_page=100&page=1
The server response was "401 Unauthorized"
Inner exception message: "You are not currently logged in."
 -> wordpress__wp_comments fetched : 0
 -> wordpress__wp_search fetched : 1623

Path: /wp-json/wp/v2/block-types?per_page=100&page=1
The server response was "401 Unauthorized"
Inner exception message: "Sorry, you are not allowed to manage block types."

Path: /wp-json/wp/v2/settings?per_page=100&page=1
The server response was "401 Unauthorized"
Inner exception message: "Sorry, you are not allowed to do that."

Path: /wp-json/wp/v2/themes?per_page=100&page=1
The server response was "400 Bad Request"
Inner exception message: "Missing parameter(s): status"

Path: /wp-json/wp/v2/plugins?per_page=100&page=1
The server response was "401 Unauthorized"
Inner exception message: "Sorry, you are not allowed to manage plugins for this site."

Path: /wp-json/wp/v2/block-directory/search?per_page=100&page=1
The server response was "400 Bad Request"
Inner exception message: "Missing parameter(s): term"
⠇ source and transform nodes
[============================]   252.884 s 128/128 100% Generating image thumbnails
[==========================  ]   160.993 s 1961/2045 96% Downloading remote files

It will just hang there on 1961/2045. I have edited my gatsby-config.js options to include these plugin options, hoping they would help:

    {
      resolve: `gatsby-source-wordpress-experimental`,
      options: {
        /*
         * The full URL of the WordPress site's GraphQL API.
         * Example : 'https://www.example-site.com/graphql'
         */
        url: `https://fakeurl.com/graphql`, //`https://wordpress-222222-33333.cloudwaysapps.com/graphql`,
        verbose: true,
        debug: {
          graphql: {
            showQueryVarsOnError: true,
          },
        },
        develop: {
          timeout: 30000,
          hardCacheMediaFiles: true,
        },
        html: {
          imageMaxWidth: process.env.NODE_ENV === `development` ? 180 : 8000,
          imageQuality: process.env.NODE_ENV === `development` ? 10 : 90,
        },
        type: {
          Post: {
            limit: process.env.NODE_ENV === `development` ? 10 : 5000,
          },
          MediaItem: {
            localFile: {
              excludeByMimeTypes: process.env.NODE_ENV === `development` ? [`video/mp4`] : []
            },
          },
        },
      },
    },

But nothing has changed. Still gets hung up at exact same spot.

I've been fighting this issue for 3 days now, unable to work on the project. I have another developer working on the project who is able to build it on his laptop. He is running a PC Ubuntu with core i5 7500. I am running on a Macbook 2018. I have no idea if the hardware is relevant but it seems to be the only variable between us. Or his internet is faster, but I have no idea. It really seems like an issue with Gatsby/WP itself.

Thanks everyone, please let me know if you know of any things I can try to get my project to build.

PS: The issue is not resolved by gatsby clean or any kind of deleting of caches or node_modules and re-installing anything.

address before stable release needs reproduction

Source

znwhite

👍4

Most helpful comment

Gahh I've run into this a few times myself, any help would be good!!

HeroshJ on 13 Oct 2020

👍3

All 30 comments

I've been having this issue as well. It'd be great to get an answer

rosborn92 on 13 Oct 2020

👍3

Gahh I've run into this a few times myself, any help would be good!!

HeroshJ on 13 Oct 2020

👍3

Hey all! Can anyone provide a reproduction /graphql endpoint (and repo possibly) I can use to reproduce?

Also the output when you run gatsby info will be helpful.

I can prioritize this higher than other things since this is a critical bug, but I need to be able to reproduce it to fix. Thanks!

TylerBarnes on 14 Oct 2020

Hi Tyler, this is what I get when I run gatsby info:

System:
    OS: macOS 10.15.5
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 12.16.3 - /usr/local/bin/node
    Yarn: 1.22.4 - /usr/local/bin/yarn
    npm: 6.14.4 - /usr/local/bin/npm
  Languages:
    Python: 2.7.16 - /usr/bin/python
  Browsers:
    Chrome: 86.0.4240.75
    Safari: 13.1.1
  npmPackages:
    gatsby: ^2.24.52 => 2.24.60
    gatsby-cli: ^2.12.90 => 2.12.97
    gatsby-image: ^2.4.16 => 2.4.18
    gatsby-plugin-antd: ^2.0.2 => 2.2.0
    gatsby-plugin-google-analytics: ^2.3.14 => 2.3.14
    gatsby-plugin-google-tagmanager: ^2.3.12 => 2.3.12
    gatsby-plugin-less: ^2.0.6 => 2.1.2
    gatsby-plugin-manifest: ^2.0.9 => 2.4.29
    gatsby-plugin-netlify-functions: ^0.0.3 => 0.0.3
    gatsby-plugin-offline: ^2.0.16 => 2.2.10
    gatsby-plugin-react-helmet: ^3.3.11 => 3.3.11
    gatsby-plugin-robots-txt: ^1.5.1 => 1.5.1
    gatsby-plugin-sass: ^2.3.12 => 2.3.13
    gatsby-plugin-sharp: ^2.6.30 => 2.6.36
    gatsby-plugin-sitemap: ^2.4.12 => 2.4.13
    gatsby-plugin-styled-components: ^3.3.10 => 3.3.11
    gatsby-source-filesystem: ^2.3.27 => 2.3.30
    gatsby-source-wordpress: ^3.3.31 => 3.3.34
    gatsby-source-wordpress-experimental: ^1.3.8 => 1.3.8
    gatsby-transformer-sharp: ^2.5.13 => 2.5.15
  npmGlobalPackages:
    gatsby-cli: 2.12.37

I will get back to you with the /graphql endpoint soon. Thank you!

znwhite on 14 Oct 2020

@TylerBarnes can I DM or email you our /graphql endpoint? Ideally we keep it private.

znwhite on 14 Oct 2020

👍2

@znwhite you can email me at [email protected] :) thanks!

TylerBarnes on 14 Oct 2020

@TylerBarnes emailed!

znwhite on 14 Oct 2020

@znwhite I have a feeling you and the other developer you're working with might be on different package versions. Can you upgrade to the latest of all Gatsby packages (or delete your yarn.lock or package-lock.json and reinstall) to see if it fixes it. There were some Gatsby core bugs around this at some point (not sure which version), but they were resolved. The latest Gatsby version is 2.24.77 for example but your project has 2.24.60 installed

TylerBarnes on 15 Oct 2020

@TylerBarnes deleted yarn.lock and package-lock.json and re-installed (making sure I upgraded to 2.24.77) and still same issue :(

znwhite on 15 Oct 2020

@znwhite can you add me to your repo? I'm not able to reproduce so I'm wondering if it's some very specific combination of things 🤔

TylerBarnes on 15 Oct 2020

@TylerBarnes Just added you with read access. Thank you!

znwhite on 15 Oct 2020

❤1

@znwhite I'm still not able to reproduce :( Can you try setting hardCacheMediaFiles to false?

TylerBarnes on 16 Oct 2020

@TylerBarnes Tried it. Still getting hung up on Downloading remote files 2010/2045.

Noticed it's getting further sometimes than others. I am not familiar enough with the underlying processes here, could this be an internet speed issue? It never finishes building when I am at office or when I am at home, but clearly this project is building sometimes for people in other locations.

Sorry if this is swaying off topic here, just really trying to pin down the problem.

We appreciate your help!

znwhite on 16 Oct 2020

@znwhite it's not swaying off topic, you're good :) Thanks for the additional info. Can you run an internet speed test so I can limit my connection and make it the same?

TylerBarnes on 16 Oct 2020

@TylerBarnes Yes, internet speed (via fast.com) is consistently coming in at 94-97 mbps.

Perhaps there is a way to limit/exclude the offending content/files? Can I limit wp_media files during development?

I have reason to believe it could be some video files that were uploaded into our WP recently. There are 34 of them. But limiting videos does not seem to work either. Latest attempt's gatsby-config.js :

{ resolve: `gatsby-source-wordpress-experimental`, options: { excludeFieldNames: [`video`], url: `https://fakeurl.com/graphql`, verbose: true, debug: { graphql: { showQueryVarsOnError: true, }, }, develop: { timeout: 30000, hardCacheMediaFiles: false, }, html: { imageMaxWidth: process.env.NODE_ENV === `development` ? 180 : 8000, imageQuality: process.env.NODE_ENV === `development` ? 10 : 90, }, type: { Post: { limit: process.env.NODE_ENV === `development` ? 5 : 5000, }, Page: { limit: process.env.NODE_ENV === `development` ? 5 : 5000, }, MediaItem: { localFile: { excludeByMimeTypes: [`video/mp4`, `video/mov`] }, }, }, }, },

znwhite on 16 Oct 2020

@znwhite currently the only way to limit media items is to limit the posts that they're attached to. Since you've limited it to 5 pages and 5 posts you're close to having limited all the media items anyway 🤔

I'll try a few more things and do a bit of brainstorming with a colleague later today and let you know what I come up with. Thanks again

TylerBarnes on 16 Oct 2020

Oh, also are you able to reproduce this if you disable gatsby-source-wordpress and just run gatsby-source-wordpress-experimental? Actually it'd be great if you could try commenting out all plugins except for gatsby-source-wordpress-experimental to make sure it's causing the issue.

TylerBarnes on 16 Oct 2020

Thinking about it a bit more, there's a good chance this is a bug with gatsby-source-wordpress or a bug where this plugin and that plugin are clashing. Because if you limit your posts to 5 in this plugin, it wont download 2k images unless those are all attached to those 10 posts/pages. So there's a high chance this bug isn't coming from the experimental plugin

TylerBarnes on 16 Oct 2020

😱 I just reproduced it! I set my internet speed exactly to yours ha! I'll dig into this a bit

TylerBarnes on 16 Oct 2020

Yep, I can confirm that this is a gatsby-source-wordpress@v3 bug. I can reliably reproduce this when I disable gatsby-source-wordpress-experimental in your gatsby-config.js (so only v3 is running). When I disable gatsby-source-wordpress I can't reproduce it. I'd recommend fully upgrading to the experimental plugin to overcome this bug! All my time right now is dedicated to making v4 stable (which will be happening very soon) so I wont have time to fix the v3 bug, especially since it'll be deprecated soon.

TylerBarnes on 16 Oct 2020

@TylerBarnes Good to know- thanks for figuring that out. But I'm noticing that when I remove gatsby-source-wordpress, I am missing all of my custom/acf content types from GraphiQL. Is there any way to get them back via gatsby-source-wordpress-experimental? The bulk of the project relies on those, not just Post/Page etc.

znwhite on 16 Oct 2020

👍1

You'll want to install https://github.com/wp-graphql/wp-graphql-acf and make sure each field group is enabled to show in GraphQL and has a field name.

TylerBarnes on 16 Oct 2020

@TylerBarnes The thing is, I believe I need gatsby-source-wordpress@v3 to query acf in the first place. We already have that plugin installed on our wordpress instance. It seems like gatsby-source-wordpress-experimental does not support acf? It does not have a useACF: true option the way that gatsby-source-wordpress@v3 does. Am I missing something?

znwhite on 16 Oct 2020

@znwhite it supports ACF if the fields are added to WPGraphQL, no need for a useAcf option. Can you confirm that you're able to query ACF data from WPGraphQL directly?

TylerBarnes on 16 Oct 2020

@TylerBarnes No, we have never been able to query ACF data directly from WPGraphQL (even with gatsby-source-wordpress).

EDIT: I lied. We were able to before, not sure why I forgot

znwhite on 16 Oct 2020

@znwhite gatsby-source-wordpress v3 uses the REST API. Make sure you follow the WPGraphQL ACF docs so that your acf field groups and fields show up in wp-graphiql, once you do that it'll show up in Gatsby automatically

TylerBarnes on 16 Oct 2020

@TylerBarnes Ok they're showing up! But the problem was because we couldn't get a build to finish... and then we upgraded our internet to be 200+ mbps and then the build finished.

We observed a strange throttling behavior in our internet download speed while Downloading remote files was running. A phone call to our internet provider bumped us up to around 250 mbps and then the build went through.

Per your observations, we are going to begin refactoring the code over to gatsby-source-wordpress-experimental queries instead of the original queries we were using.

Thank you for all your help!

znwhite on 16 Oct 2020

❤1 🎉1

Great to hear! :D

TylerBarnes on 16 Oct 2020

🚀1

@TylerBarnes any updates here?