Gatsby: Gatsby fails resolving node_modules in monorepo

[josefaidt]

Description

When using a monorepo with lerna and Yarn workspaces I am attempting to set up multiple, different Gatsby sites under a ./starters folder (from scratch or using the cli), and they will all use a shared component library under ./packages folder. The issue seems to be affecting Gatsby itself and how it resolves packages.

I am able to use my existing React stack setup in the same monorepo without issue. Gatsby site fails with and without react and react-dom in root package.json

SKIP TO SOLUTION for using base Gatsby in a monorepo

Steps to reproduce

• Setting up gatsby from scratch
  
  
  
  1. mkdir starters/gatsby
  
  
  2. cd starters/gatsby
  
  
  3. init blank project: yarn -y init
  
  
  4. install my dependencies: yarn workspace gatsby add gatsby gatsby-image gatsby-plugin-manifest gatsby-plugin-offline gatsby-plugin-react-helmet gatsby-plugin-sharp gatsby-plugin-styled-components gatsby-source-filesystem gatsby-transformer-sharp prop-types styled-components react react-dom react-helmet
  
  
  5. set up build and develop scripts
  
  
  6. Run start script. Everything runs great on the first run with 0 files added (no gatsby-* files, no src directory).
  
  
  7. Quit running script, add gatsby-config.js and src directory from default starter
  
  
  8. Run develop script. Fails.
  
  
• Setting up gatsby site via cli
  
  
  
  1. cd starters
  
  
  2. gatsby new new-site
  
  
  3. Change package.json name to "gatsby" and run: cd .. && yarn workspace gatsby run develop, site runs fine
  
  
  4. add my component library from packages/
  
  
  5. lerna clean && lerna bootstrap
  
  
  6. Run the site again using yarn workspace gatsby run develop and it fails finding the module gatsby/graphql
  
  
• After site is built, functional, and all package dependencies have been bootstrapped
  
  
  
  1. add a new plugin, yarn workspace gatsby add gatsby-transformer-json
  
  
  2. modify the gatsby-config.js file as necessary
  
  
  3. clear the site's cache directory
  
  
  4. Run the site using yarn workspace gatsby run develop
  
  
  5. Site fails to resolve new module (see stack trace below)
  
  

Scripts

{
  "scripts": {
    "start": "yarn develop",
    "develop": "gatsby develop --verbose",
    "build": "gatsby build"
  }
}

Expected result

Running yarn workspace gatsby run develop runs the site as intended. Sometimes this actually works after freshly installing new gatsby site to my starters folder. I can quit, bootstrap my dependencies and rerun the same command to expect the failure describe below (without gatsby as a dependency in the root package.json).

Actual result

Running yarn workspace gatsby run develop Gatsby fails on resolving only one of three packages:

• gatsby/graphql (most common)
• gatsby-plugin-react-helmet
• gatsby-source-filesystem
• gatsby-transformer-json

Stack Traces

Stack Trace for gatsby-plugin-react-helmet

verbose 0.343 set gatsby_log_level: "verbose"
verbose 0.346 set gatsby_executing_command: "develop"
verbose 0.346 loading local command from: /Users/josef/Documents/playground/{ROOT_PROJECT_FOLDER}/node_modules/gatsby/dist/commands/develop.js
verbose 1.584 running command: develop
success open and validate gatsby-configs — 0.009 s
error Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you need to install its package?

Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n  eed to install its package?

- load.js:102 resolvePlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

- load.js:114 processPlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

- load.js:164 config.plugins.forEach.plugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

- Array.forEach

- load.js:163 module.exports
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

- index.js:56 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

- Generator.next

- debuggability.js:313 Promise._execute
  [widget-library]/[bluebird]/js/release/debuggability.js:313:9

- promise.js:483 Promise._resolveFromExecutor
  [widget-library]/[bluebird]/js/release/promise.js:483:18

- promise.js:79 new Promise
  [widget-library]/[bluebird]/js/release/promise.js:79:10

- index.js:96 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

- index.js:124 
  [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36

error UNHANDLED REJECTION

Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n  eed to install its package?

- load.js:102 resolvePlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

- load.js:114 processPlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

- load.js:164 config.plugins.forEach.plugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

- Array.forEach

- load.js:163 module.exports
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

- index.js:56 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

- Generator.next

- debuggability.js:313 Promise._execute
  [widget-library]/[bluebird]/js/release/debuggability.js:313:9

- promise.js:483 Promise._resolveFromExecutor
  [widget-library]/[bluebird]/js/release/promise.js:483:18

- promise.js:79 new Promise
  [widget-library]/[bluebird]/js/release/promise.js:79:10

- index.js:96 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

- index.js:124 
  [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1

Stack Trace for gatsby/graphl

yarn workspace v1.13.0
yarn run v1.13.0
$ gatsby develop
gatsby develop

Start development server. Watches files, rebuilds, and hot reloads if
something changes

Options:
  --verbose                   Turn on verbose output
                                                [boolean] [default: false]
  --no-color                  Turn off the color in output
                                                [boolean] [default: false]
  -H, --host                  Set host. Defaults to localhost
                                           [string] [default: "localhost"]
  -p, --port                  Set port. Defaults to 8000
                                                [string] [default: "8000"]
  -o, --open                  Open the site in your (default) browser for
                              you.                               [boolean]
  -S, --https                 Use HTTPS. See
                              https://www.gatsbyjs.org/docs/local-https/
                              as a guide                         [boolean]
  -c, --cert-file             Custom HTTPS cert file (relative path; also
                              required: --https, --key-file). See
                              https://www.gatsbyjs.org/docs/local-https/
                                                    [string] [default: ""]
  -k, --key-file              Custom HTTPS key file (relative path; also
                              required: --https, --cert-file). See
                              https://www.gatsbyjs.org/docs/local-https/
                                                    [string] [default: ""]
  --open-tracing-config-file  Tracer configuration file (open tracing
                              compatible). See
                              https://www.gatsbyjs.org/docs/performance-tr
                              acing/                              [string]
  -h, --help                  Show help                          [boolean]
  -v, --version               Show version number                [boolean]

error There was a problem loading the local develop command. Gatsby may not be installed. Perhaps you need to run "npm install"?


  Error: Cannot find module 'gatsby/graphql'

  - loader.js:611 Function.Module._resolveFilename
    internal/modules/cjs/loader.js:611:15

  - loader.js:537 Function.Module._load
    internal/modules/cjs/loader.js:537:25

  - loader.js:665 Module.require
    internal/modules/cjs/loader.js:665:17

  - v8-compile-cache.js:159 require
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - index.js:8 Object.<anonymous>
    [widget-library]/[babel-plugin-remove-graphql-queries]/index.js:8:17

  - v8-compile-cache.js:178 Module._compile
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:747 Object.Module._extensions..js
    internal/modules/cjs/loader.js:747:10

  - loader.js:628 Module.load
    internal/modules/cjs/loader.js:628:32

  - loader.js:568 tryModuleLoad
    internal/modules/cjs/loader.js:568:12

  - loader.js:560 Function.Module._load
    internal/modules/cjs/loader.js:560:3

  - loader.js:665 Module.require
    internal/modules/cjs/loader.js:665:17

  - v8-compile-cache.js:159 require
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - file-parser.js:21 Object.<anonymous>
    [gatsby]/[gatsby]/dist/internal-plugins/query-runner/file-parser.js:21    :23

  - v8-compile-cache.js:178 Module._compile
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:747 Object.Module._extensions..js
    internal/modules/cjs/loader.js:747:10

  - loader.js:628 Module.load
    internal/modules/cjs/loader.js:628:32


error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node
Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run develop
Directory: /Users/josef/Documents/playground/{PROJECT}/starters/gatsby
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

Stack trace for gatsby-transformer-json

yarn workspace v1.13.0
yarn run v1.13.0
$ yarn develop
$ gatsby develop
success open and validate gatsby-configs — 0.008 s
error Unable to find plugin "gatsby-transformer-json". Perhaps you need to install its package?


  Error: Unable to find plugin "gatsby-transformer-json". Perhaps you need   to install its package?

  - load.js:102 resolvePlugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

  - load.js:114 processPlugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

  - load.js:164 config.plugins.forEach.plugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

  - Array.forEach

  - load.js:163 module.exports
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

  - index.js:56 
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

  - Generator.next

  - debuggability.js:313 Promise._execute
    [widget-library]/[bluebird]/js/release/debuggability.js:313:9

  - promise.js:483 Promise._resolveFromExecutor
    [widget-library]/[bluebird]/js/release/promise.js:483:18

  - promise.js:79 new Promise
    [widget-library]/[bluebird]/js/release/promise.js:79:10

  - index.js:96 
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

  - index.js:124 
    [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36


error UNHANDLED REJECTION


  Error: Unable to find plugin "gatsby-transformer-json". Perhaps you need   to install its package?

  - load.js:102 resolvePlugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

  - load.js:114 processPlugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

  - load.js:164 config.plugins.forEach.plugin
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

  - Array.forEach

  - load.js:163 module.exports
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

  - index.js:56 
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

  - Generator.next

  - debuggability.js:313 Promise._execute
    [widget-library]/[bluebird]/js/release/debuggability.js:313:9

  - promise.js:483 Promise._resolveFromExecutor
    [widget-library]/[bluebird]/js/release/promise.js:483:18

  - promise.js:79 new Promise
    [widget-library]/[bluebird]/js/release/promise.js:79:10

  - index.js:96 
    [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

  - index.js:124 
    [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36


error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node
Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run start
Directory: /Users/josef/Documents/playground/{PROJECT}/starters/gatsby
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

Environment

  System:
    OS: macOS 10.14.3
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Shell: 2.7.1 - /usr/local/bin/fish
  Binaries:
    Node: 11.8.0 - ~/.nvm/versions/node/v11.8.0/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.5.0 - ~/.nvm/versions/node/v11.8.0/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 72.0.3626.96
    Firefox: 64.0
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.0 => 2.1.0 

SOLUTION (well, was)

After much testing and trial/error, I've found the monorepo needs gatsby declared in the root package.json

• Once added, run lerna clean && lerna bootstrap
• Then run the site, yarn workspace gatsby run develop

I typed this up and continued working thinking it may be my environment, however after adding another plugin the site began to break and show similar symptoms to those described above.

[josefaidt]

Actually this may be my environment. When running yarn workspace gatsby add gatsby-transformer-json the dependency gets added to the correct package.json but the dependency does not actually get installed.

Does anyone have any insight to why this may be happening?

[josefaidt]

I've exhausted myself trying to figure this one out. I don't think this has to deal directly with Gatsby or its package, but I do think it's odd that I'm having issues with only a handful of plugins. The bulk of the issue has been resolve by adding gatsby as a dependency in the project root (not sure why).

The plugins I am having issues with:

• gatsby-transformer-json
• gatsby-transformer-remark

I'm going to leave this issue open for now in case anyone has any insight

[pieh]

This is very weird, because folks working on some features use yarn workspaces quite often and didn't seem to hit problems with it. It seems like workspaces isn't correctly installing deps and setting symlinks? Maybe issue lies with yarn itself not correctly setting this up? Did you try updating/downgrading it? Also did you try to verify how node_modules in site itself look like - do they have those packages installed / symlinked?

[josefaidt]

I agree this is very odd, and I thought I set everything up to the spec. I'm removing lerna from the equation to see if the issue persists. The dependencies weren't being installed at all. They are added to the root's lockfile and the package's package.json file, but the package was nowhere to be found in any node_modules folders.

I'm going to play around with this some more after learning about the nohoist workspace option.

[josefaidt]

@pieh

Ok, I ended up getting this working. Using @jedrichards snippet from his monorepo example:

"workspaces": {
    "nohoist": [
      "gatsby"
    ]
  }

I didn't know the nohoist option existed as it is not mentioned on the official yarn documentation but in a yarn blog post. BUT this didn't work for me given my directory structure:

.
├── README.md
├── components
│   ├── package.json
│   └── src
│       ├── SharedComponent
│       └── index.js
├── package.json
├── sites
│   ├── gatsby-app
│   │   ├── LICENSE
│   │   ├── README.md
│   │   ├── gatsby-browser.js
│   │   ├── gatsby-config.js
│   │   ├── gatsby-node.js
│   │   ├── gatsby-ssr.js
│   │   ├── node_modules
│   │   ├── package.json
│   │   ├── public
│   │   └── src
│   └── gatsby-app2
│       ├── LICENSE
│       ├── README.md
│       ├── gatsby-browser.js
│       ├── gatsby-config.js
│       ├── gatsby-node.js
│       ├── gatsby-ssr.js
│       ├── node_modules
│       ├── package.json
│       └── public
└── yarn.lock

The trick was to use a globbing pattern in the nohoist array like so,

"workspaces": {
    "nohoist": [
      "**/gatsby",
      "gatsby/**",
      "**/gatsby-*",
      "gatsby-*/**",
      "**/*-gatsby",
      "*-gatsby/**"
    ]
  }

I feel like this definitely needs to be mentioned in the Gatsby docs.

[DSchau]

@josefaidt I'm still kind of unsure _why_ there's an issue there. That nohoist looks pretty gnarly. I think in this case it probably makes sense to investigate what exactly breaks with that particular directory structure rather than documenting this scenario--which seems a bit niche-y and may just lead to confusion.

For instance, I structured a recent repository I created as a monorepo with Yarn workspaces, and I didn't really have any issues whatsoever with resolving Gatsby. https://github.com/DSchau/gatsby-drupal-workshop

Is there an easily discernible difference or something we can fix here? If at all possible, sharing a repository, rather than documenting its structure, would be super helpful!

[josefaidt]

At first I thought this was an issue with Gatsby and how it resolves modules as I did not have an issue with a pure React build. That is why I raised the issue, however I now know that it most likely is not a Gatsby-specific issue, but rather a "quirk", if you will, to using it this way.

I took a look at your repo and I'm honestly surprised you haven't had any issues without any hoisting. But what was weird was that originally my issue was affecting specific gatsby packages.

I'm still not quite sure what the root cause of the issue is, and unfortunately I cannot share the original repo but I made a proof of concept: https://github.com/josefaidt/monorepo-gatsby

Considering this issue is resolved and in the end not entirely relevant to Gatsby I am closing it.

[jedrichards]

@josefaidt There were a few tiny errors in your proof of concept above. Check it out: https://github.com/josefaidt/monorepo-gatsby/pull/1

[carllippert]

@josefaidt did this proof of concept work? I am trying to get a lerna monorepo set up and am running into about a days worth of issues seemingly related to staticquery's not being picked up if they are not in main src folder. In demo i see you had all your quries in site folder but was wondering if you went further with this?

[josefaidt]

@carllippert yes, however I use yarn without lerna now. My personal site uses a monorepo with a theme, demo site, and app site. In app/content/pages/resume.mdx you can see I'm using a page query outside of the source folder

[carllippert]

@josefaidt did removing lerna aid in this? I would have to do a lot of digging but i think when you use the page-creator plugin it then can pickup pageQueries but yet basically just scans src folder for StaticQuery and useStaticQueryHooks thus killing any components say in a shared ui package. I use page creator to generate pages in a dif package and that seemingly works its just the underlying components on the page. Anyways thanks for giving me something to look over for inspiration. 👍

[timbakkum]

I had a similar issue in a monorepo with a gatsby project in a seperate package: storybook with a custom webpack config using file-loader in one package and gatsby project in another package. Gatsby requires v 1.1.11 of file-loader while the other packages use v 3 or higher.
Yarn put 1 version of file-loader in the storybook package's node_modules but did not put/hoist a version in the global node_modules, therefor the gatsby package could not find file-loader.

Using the fix described by @josefaidt gatsby is able to resolve file-loader again (when importing local fonts in my case). I don't know if this issue lies with yarn workspaces or with the way gatsby dependencies are listed/required, but hopefully this sheds some more light on it for somebody.