Gatsby: Setup in monorepo (`gatsby` dependency is required?)

[stephanschubert]

I have a monorepo set up with lerna (which shouldn't matter) and moved all (dev)dependencies into the root package.json. But it is required that gatsby is added to the package's own package.json.

I'm not running npm i inside the (sub-)package—I just added it.

Are there any (legacy?) reasons why this check is required or could it be removed/replaced with a more flexible one?

[pieh]

If package (site in case) uses dependency, then it should declare that dependency. If you want to avoid manually installing dependencies in each individual package please take a look at lerna bootstrap that will handle that. Another useful thing to consider is using yarn workspaces.

Gatsby doesn't really require anything, so there is no other way to determine if current working directory is actually root of gatsby project - if we would remove that check you could run gatsby build/develop anywhere and it would probably build empty site which is definitely not great DX

[stephanschubert]

I don't use lerna bootstrap nor yarn workspaces for different reasons. (FYI: Moving all deps into root package.json allows me to just run npm i packages/* to setup everything up.)

But back to topic: The current check does not ensure the current directory is actually a valid (as in buildable) Gatsby project. A lot of things can be wrong or missing in any case..?

[sidharthachatterjee]

Like @pieh mentioned, Gatsby doesn't really require anything so the current check for gatsby as a dependency or dev dependency makes sense because it is a pretty strong indicator that the current folder _is_ a Gatsby project if it depends on it.

If I understand correctly, you want to avoid manually installing dependencies for every individual package (for which you're using npm i packages/*)

lerna bootstrap does the same thing:

npm install all external dependencies of each package.
Symlink together all Lerna packages that are dependencies of each other.
npm run prepublish in all bootstrapped packages.
npm run prepare in all bootstrapped packages.

On a separate note, it is a good practice for individual packages to list their own dependencies so that they are self contained and can work in isolation.

I'm closing this issue for now but please feel free to reopen if there is anything else.

[stephanschubert]

@sidharthachatterjee Yes, that's what I want(ed) to avoid. Just for the record: "my" approach is a suggestion from lerna maintainers.

Besides that moving all deps into one root package.json avoids issues with different versions. Personally I take the tradeoff of having to put the deps back when publishing.

Anyway, I can live with this check but wanted to raise the question whether it's still useful.

[pieh]

You can also skip using gatsby-cli and call build/develop commands directly with some small custom wrapper:
here's entry point for build https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby/src/commands/build.js - you can write very small js file to setup BuildArgs (that normally are setup by cli)

I do think that tradeoff for not checking local package.json is not worth supporting this pretty niche use case - that's the only sanity check we can do :/

[sidharthachatterjee]

Thank you for bringing it up anyway, @stephanschubert

If there's more interest in the usecase, we'll relook at this in the future. In the mean time, happy building 🙂

[stephanschubert]

@pieh Thanks for the hint, I will look into that.

[josefaidt]

Hi all, after much trial and error getting a similar setup running I noticed without declaring gatsby in the root package.json file gatsby sites in a monorepo will not run - failing resolving modules such as gatsby/graphql and gatsby-plugin-react-helmet.

I have a whole issue typed up (see below) but figured I could tack on to this issue. I did not see this note in the documentation, and was wondering if it could be added somewhere?

My issue summary

Description

When using a monorepo with lerna and Yarn workspaces I am attempting to set up multiple, different Gatsby sites under a ./starters folder (from scratch or using the cli), and they will all use a shared component library under ./packages folder. The issue seems to be affecting Gatsby itself and how it resolves hoisted packages.

I am able to use my existing React stack setup in the same monorepo without issue. Gatsby site fails with and without react and react-dom in root package.json

SKIP TO SOLUTION

Steps to reproduce

• Setting up gatsby from scratch
  
  
  
  1. mkdir starters/gatsby
  
  
  2. cd starters/gatsby
  
  
  3. init blank project: yarn -y init
  
  
  4. install my dependencies: yarn workspace gatsby add gatsby gatsby-image gatsby-plugin-manifest gatsby-plugin-offline gatsby-plugin-react-helmet gatsby-plugin-sharp gatsby-plugin-styled-components gatsby-source-filesystem gatsby-transformer-sharp prop-types styled-components react react-dom react-helmet
  
  
  5. set up build and develop scripts
  
  
  6. Run start script. Everything runs great on the first run with 0 files added (no gatsby-* files, no src directory).
  
  
  7. Quit running script, add gatsby-config.js and src directory from default starter
  
  
  8. Run develop script. Fails.
  
  
• Setting up gatsby site via cli
  
  
  
  1. cd starters
  
  
  2. gatsby new new-site
  
  
  3. Change package.json name to "gatsby" and run: cd .. && yarn workspace gatsby run develop, site runs fine
  
  
  4. add my component library from packages/
  
  
  5. lerna clean && lerna bootstrap
  
  
  6. Run the site again using yarn workspace gatsby run develop and it fails finding the module gatsby/graphql
  
  

Scripts

{
  "scripts": {
    "start": "yarn develop",
    "develop": "gatsby develop --verbose",
    "build": "gatsby build"
  }
}

Expected result

Running yarn workspace gatsby run develop runs the site as intended. Sometimes this actually works after freshly installing new gatsby site to my starters folder. I can quit, bootstrap my dependencies and rerun the same command to expect the failure describe below (without gatsby as a dependency in the root package.json).

Actual result

Running yarn workspace gatsby run develop Gatsby fails on resolving only one of three packages:

• gatsby/graphql (most common)
• gatsby-plugin-react-helmet
• gatsby-source-filesystem

Specifically:

Stack Trace for gatsby-plugin-react-helmet

verbose 0.343 set gatsby_log_level: "verbose"
verbose 0.346 set gatsby_executing_command: "develop"
verbose 0.346 loading local command from: /Users/{USER}/Documents/{ROOT_PROJECT_FOLDER}/node_modules/gatsby/dist/commands/develop.js
verbose 1.584 running command: develop
success open and validate gatsby-configs — 0.009 s
error Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you need to install its package?

Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n  eed to install its package?

- load.js:102 resolvePlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

- load.js:114 processPlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

- load.js:164 config.plugins.forEach.plugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

- Array.forEach

- load.js:163 module.exports
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

- index.js:56 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

- Generator.next

- debuggability.js:313 Promise._execute
  [widget-library]/[bluebird]/js/release/debuggability.js:313:9

- promise.js:483 Promise._resolveFromExecutor
  [widget-library]/[bluebird]/js/release/promise.js:483:18

- promise.js:79 new Promise
  [widget-library]/[bluebird]/js/release/promise.js:79:10

- index.js:96 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

- index.js:124 
  [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36

error UNHANDLED REJECTION

Error: Unable to find plugin "gatsby-plugin-react-helmet". Perhaps you n  eed to install its package?

- load.js:102 resolvePlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:102:11

- load.js:114 processPlugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:114:20

- load.js:164 config.plugins.forEach.plugin
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:164:20

- Array.forEach

- load.js:163 module.exports
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/load.js:163:20

- index.js:56 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:56:21

- Generator.next

- debuggability.js:313 Promise._execute
  [widget-library]/[bluebird]/js/release/debuggability.js:313:9

- promise.js:483 Promise._resolveFromExecutor
  [widget-library]/[bluebird]/js/release/promise.js:483:18

- promise.js:79 new Promise
  [widget-library]/[bluebird]/js/release/promise.js:79:10

- index.js:96 
  [widget-library]/[gatsby]/dist/bootstrap/load-plugins/index.js:96:17

- index.js:124 
  [widget-library]/[gatsby]/dist/bootstrap/index.js:124:36

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1

Stack Trace for gatsby/graphl

yarn workspace v1.13.0
yarn run v1.13.0
$ gatsby develop
gatsby develop

Start development server. Watches files, rebuilds, and hot reloads if
something changes

Options:
  --verbose                   Turn on verbose output
                                                [boolean] [default: false]
  --no-color                  Turn off the color in output
                                                [boolean] [default: false]
  -H, --host                  Set host. Defaults to localhost
                                           [string] [default: "localhost"]
  -p, --port                  Set port. Defaults to 8000
                                                [string] [default: "8000"]
  -o, --open                  Open the site in your (default) browser for
                              you.                               [boolean]
  -S, --https                 Use HTTPS. See
                              https://www.gatsbyjs.org/docs/local-https/
                              as a guide                         [boolean]
  -c, --cert-file             Custom HTTPS cert file (relative path; also
                              required: --https, --key-file). See
                              https://www.gatsbyjs.org/docs/local-https/
                                                    [string] [default: ""]
  -k, --key-file              Custom HTTPS key file (relative path; also
                              required: --https, --cert-file). See
                              https://www.gatsbyjs.org/docs/local-https/
                                                    [string] [default: ""]
  --open-tracing-config-file  Tracer configuration file (open tracing
                              compatible). See
                              https://www.gatsbyjs.org/docs/performance-tr
                              acing/                              [string]
  -h, --help                  Show help                          [boolean]
  -v, --version               Show version number                [boolean]

error There was a problem loading the local develop command. Gatsby may not be installed. Perhaps you need to run "npm install"?


  Error: Cannot find module 'gatsby/graphql'

  - loader.js:611 Function.Module._resolveFilename
    internal/modules/cjs/loader.js:611:15

  - loader.js:537 Function.Module._load
    internal/modules/cjs/loader.js:537:25

  - loader.js:665 Module.require
    internal/modules/cjs/loader.js:665:17

  - v8-compile-cache.js:159 require
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - index.js:8 Object.<anonymous>
    [widget-library]/[babel-plugin-remove-graphql-queries]/index.js:8:17

  - v8-compile-cache.js:178 Module._compile
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:747 Object.Module._extensions..js
    internal/modules/cjs/loader.js:747:10

  - loader.js:628 Module.load
    internal/modules/cjs/loader.js:628:32

  - loader.js:568 tryModuleLoad
    internal/modules/cjs/loader.js:568:12

  - loader.js:560 Function.Module._load
    internal/modules/cjs/loader.js:560:3

  - loader.js:665 Module.require
    internal/modules/cjs/loader.js:665:17

  - v8-compile-cache.js:159 require
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - file-parser.js:21 Object.<anonymous>
    [gatsby]/[gatsby]/dist/internal-plugins/query-runner/file-parser.js:21    :23

  - v8-compile-cache.js:178 Module._compile
    [widget-library]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:747 Object.Module._extensions..js
    internal/modules/cjs/loader.js:747:10

  - loader.js:628 Module.load
    internal/modules/cjs/loader.js:628:32


error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed.
Exit code: 1
Command: /Users/josef/.nvm/versions/node/v11.8.0/bin/node
Arguments: /usr/local/Cellar/yarn/1.13.0/libexec/lib/cli.js run develop
Directory: /Users/josef/Documents/playground/widget-library/starters/gatsby
Output:

info Visit https://yarnpkg.com/en/docs/cli/workspace for documentation about this command.

Environment

  System:
    OS: macOS 10.14.3
    CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
    Shell: 2.7.1 - /usr/local/bin/fish
  Binaries:
    Node: 11.8.0 - ~/.nvm/versions/node/v11.8.0/bin/node
    Yarn: 1.13.0 - /usr/local/bin/yarn
    npm: 6.5.0 - ~/.nvm/versions/node/v11.8.0/bin/npm
  Languages:
    Python: 2.7.10 - /usr/bin/python
  Browsers:
    Chrome: 72.0.3626.96
    Firefox: 64.0
    Safari: 12.0.3
  npmPackages:
    gatsby: ^2.1.0 => 2.1.0 

SOLUTION

After much testing and trial/error, I've found the monorepo needs gatsby declared in the root package.json

• Once added, run lerna clean && lerna bootstrap
• Then run the site, yarn workspace gatsby run develop