Next.js: 9.5.0 introduced compile error / breaks when using the pnpm package manager
Bug report
Upgrading a hello world project from 9.4.4 to 9.5.0 when using pnpm introduces a compile error for next build (see below)
To Reproduce
install pnpm on your system.
clone example repo https://github.com/bitfrost/example_pnpm_next_issue
cd into repo
run:
pnpm install
pnpm next build
Get
next "build"
Creating an optimized production build
Compiled successfully.
Automatically optimizing pages ..
Error occurred prerendering page "/404". Read more: https://err.sh/next.js/prerender-error
Error: Minified React error #321; visit https://reactjs.org/docs/error-decoder.html?invariant=321 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Z (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1028:404)
at module.exports.mbhs.exports.useContext (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1033:261)
at Html (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:257:29)
at d (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:36:498)
at $a (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:16)
at a.b.render (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:476)
at a.b.read (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:18)
at renderToStaticMarkup (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:54:462)
at renderDocument (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next/dist/next-server/server/render.js:3:594)
at renderToHTML (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next/dist/next-server/server/render.js:47:72)
Error occurred prerendering page "/". Read more: https://err.sh/next.js/prerender-error
Error: Minified React error #321; visit https://reactjs.org/docs/error-decoder.html?invariant=321 for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
at Z (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1028:404)
at Object.module.exports.mbhs.exports.useState (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:1034:277)
at Link (/Users/kobrien/dev/nx-5/.next/server/pages/index.js:1975:50)
at d (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:36:498)
at $a (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:39:16)
at a.b.render (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:476)
at a.b.read (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:44:18)
at renderToString (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-dom-server.node.production.min.js:54:364)
at Object.renderPage (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next/dist/next-server/server/render.js:45:686)
at Function.getInitialProps (/Users/kobrien/dev/nx-5/.next/server/pages/_document.js:222:19)
Build error occurred
Error: Export encountered errors on following paths:
/
/404
at exportApp (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next/dist/export/index.js:24:1103)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at async build (/Users/kobrien/dev/nx-5/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/next/dist/build/index.js:37:218)
Automatically optimizing pages . ERROR  Command failed with exit code 1.
Move dependency to "next": "9.4.4" in package.json
repeat:
pnpm install
pnpm next build
project works and builds
Expected behavior
pnpm and be used with a minimal next.js app
System information
OS: macos
Version of Next.js: 9.5.0
- Version of Node.js: 12.6.3
bitfrost
All 18 comments
~Seeing the same thing, but on regular npm~ - nvm
IanMitchell
on 29 Jul 2020
Interesting @IanMitchell . With the above example repo, I don't see this using npm (6.14.4).
Can you confirm my example repo doesn't work with you for npm (and if so what version of npm / node?) (Look out for stray node_modules/ in parent directories, i have seen those cause similar webpack errors around duplicate 'react/package versions'). Locally I can use the example repo and npm to build and run via next without any issues. Tested on 2 machines.
bitfrost
on 29 Jul 2020
Whoops, you're right - I wasn't seeing the same thing, I was getting a different bug covered by an existing report / PR. My bad for not reading the stack trace carefully enough, sorry!
IanMitchell
on 29 Jul 2020
I can repo with 9.4.5-canary.8 (9.4.5-canary.7 issue is not present)
I will dig through these https://github.com/vercel/next.js/compare/v9.4.5-canary.7...v9.4.5-canary.8
bitfrost
on 29 Jul 2020
Via some bi-secting This appears to be introduced by
https://github.com/vercel/next.js/commit/e125d905a0dd93d247c6122d349c2c90268f0713
@timneutkens . I am not grokking why yet, but will dig a bit more tomorrow.
bitfrost
on 29 Jul 2020
removing the use of this hook gets rid of the 'you may be violating the rules of hooks' error. I tweaked (_document.tsx)
// const { inAmpMode } = useContext(DocumentComponentContext)._documentProps
const inAmpMode = true; <-- not a real fix, e125d90 did introduce using the hook versus a class component
The above hack lead me to this apparently bad import
https://github.com/vercel/next.js/blob/canary/packages/next/next-server/server/lib/path-match.ts#L1
notice the direct import from compiled/dist/
import * as pathToRegexp from 'next/dist/compiled/path-to-regexp'
This package is missing here:
https://github.com/vercel/next.js/tree/canary/packages/next/compiled
changing path-match.ts and removing the use of a hook in _document.tsx
fixes things when using pnpm
Change:
import * as pathToRegexp from 'path-to-regexp' + package.json update adding 'path-to-regexp'
Not sure yet why I am getting the hook error only from pnpm.
I will dig more tomorrow potentially raise a PR
bitfrost
on 29 Jul 2020
I've unmarked this as p0 because this appears to be a bug with pnpm. pnpm is duplicating copies of react and react-dom which causes three instances to be loaded at build-time:
/example_pnpm_next_issue/node_modules/react/index.js
/example_pnpm_next_issue/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/react/index.js
/.pnpm/[email protected][email protected]/node_modules/react/index.js
Timer
on 30 Jul 2020
Marking this as a duplicate of #9022
Timer
on 30 Jul 2020
@Timer I followed the following procedure and found the same instance of react. pnpm points things via symlinks links to single shared versions of these packages. Can you share how you produced the above snippet?
Using the facebook suggested method, I got a 'true' using the below:
https://reactjs.org/warnings/invalid-hook-call-warning.html
// Add this in node_modules/react-dom/index.js
window.React1 = require('react');
// Add this in your component file
require('react-dom');
window.React2 = require('react');
console.log(window.React1 === window.React2);
Same version of pnpm, this works fine previous to this change.
bitfrost
on 30 Jul 2020
This does feel like a pnpm bug , the work around here fixes my example repo
https://github.com/pnpm/pnpm/issues/2695
bitfrost
on 30 Jul 2020
pnpm is duplicating copies of react and react-dom which causes three instances to be loaded at build-time
I don't see duplicated.
This
/.pnpm/[email protected][email protected][email protected]/node_modules/react/
and this
/.pnpm/[email protected][email protected]/node_modules/react/
are just symlinks to the same directory:
/.pnpm/[email protected]/node_modules/react
Seems like a bug in React. React should resolve these symlinks to their real locations. That's how node's resolution algorithm works.
zkochan
on 1 Aug 2020
I agree @zkochan not pnpm bug, all same copies of React. While the workaround 'worked', it should not be needed at all.
bitfrost
on 4 Aug 2020
tl;dr resolveRequest should be updated to follow symlinks, which I believe should fix this issue.
Did some debugging; here's what I found.
The issue is in this handleExternals function where Next.js determines which modules webpack should include in the server bundle (written to .next/server/pages/_document.js) and which should be treated as "external", i.e. require()'d by Node.js at run time. We end up with two instances of the 'react' module because one is included in the bundle and one is require()'d by Node.js.
I added some log statements to this function and ran next dev:
with npm
> [email protected] dev /Users/spencerelliott/Dev/elliottsj/my-app
> next dev
ready - started server on http://localhost:3000
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/next/dist/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/next/dist/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
with pnpm
> [email protected] dev /Users/spencerelliott/Dev/elliottsj/my-app
> next dev
ready - started server on http://localhost:3000
context /Users/spencerelliott/Dev/elliottsj/my-app/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
externalizing module
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pages
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected][email protected]/node_modules/styled-jsx/dist
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected][email protected]/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
context /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected][email protected]/node_modules/styled-jsx/dist
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected][email protected]/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
baseRes !== res. bundling module.
Using npm, we get our desired result: the 'react' module is always externalized.
Using pnpm, 'react' is only externalized when the importer is in the /Users/spencerelliott/Dev/elliottsj/my-app/pages directory (aka the context). From these other contexts, React is bundled instead:
node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pagesnode_modules/.pnpm/[email protected][email protected]/node_modules/styled-jsx/dist
This happens because handleExternals performs a check to make sure a module is resolved to the same location from both the context and the project directory dir before externalizing it:
If baseRes !== res, then the module is forced to be bundled.
If we take this example:
res /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/react/index.js
baseRes /Users/spencerelliott/Dev/elliottsj/my-app/node_modules/react/index.js
Both of these paths actually resolve to the same location on disk:
$ cd node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/react
$ pwd -P
/Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]/node_modules/react
$ cd -
$ cd node_modules/react
$ pwd -P
/Users/spencerelliott/Dev/elliottsj/my-app/node_modules/.pnpm/[email protected]/node_modules/react
So, I think the root cause of this issue is that the resolveRequest function does not follow symlinks. If it followed symlinks, then baseRes would always equal res, so React would be externalized as we expect.
elliottsj
on 16 Aug 2020
I made a pull request which updates the behaviour of resolveRequest so it follows symlinks: https://github.com/vercel/next.js/pull/16369
This fixes the problem described in this issue.
However it does not resolve the issue which this is marked as a duplicate of: https://github.com/vercel/next.js/issues/9022. Importing React components from outside of the project folder is a slightly different issue due to the fact that you can have different copies of the React package on disk.
Appreciate if a maintainer can re-open this issue.
elliottsj
on 24 Aug 2020
Would love to see #16369 merged as well. We are currently on a pnpm monorepo and this issue is preventing us from upgrading to 9.5 from 9.4. #9022 seems to be a slew of issues where duplicate react is a symptom and not necessarily related to this specific issue that @elliottsj has root caused so I too would appreciate if the maintainers can perhaps re-open this and consider #16369.
keyworks
on 29 Aug 2020
@elliottsj I missed this PR and created a similar one (but made it pnpm specific) and attached it to
https://github.com/vercel/next.js/issues/16471 as well. I would be thrilled if one of them is merged. In the mean time I postinstall the patched files into next.js (ick)
bitfrost
on 30 Aug 2020
@bitfrost What you mean with postinstall?
zefexdeveloper
on 30 Aug 2020
In my own projects, I setup something like:
"scripts": {
"postinstall": "a script to copy files into node_modues/next/dist with my patched PR files"
}
so post pnpm install , next.js gets 'fixed' and I can continue to work using pnpm + next.
bitfrost
on 30 Aug 2020
Related issues
havefive
·
3Comments
wagerfield
·
3Comments
rauchg
·
3Comments
lixiaoyan
·
3Comments
swrdfish
·
3Comments
Most helpful comment
tl;dr
resolveRequestshould be updated to follow symlinks, which I believe should fix this issue.Did some debugging; here's what I found.
The issue is in this handleExternals function where Next.js determines which modules webpack should include in the server bundle (written to
.next/server/pages/_document.js) and which should be treated as "external", i.e.require()'d by Node.js at run time. We end up with two instances of the'react'module because one is included in the bundle and one isrequire()'d by Node.js.I added some log statements to this function and ran
next dev:with npm
with pnpm
Using npm, we get our desired result: the
'react'module is always externalized.Using pnpm,
'react'is only externalized when the importer is in the/Users/spencerelliott/Dev/elliottsj/my-app/pagesdirectory (aka thecontext). From these other contexts, React is bundled instead:node_modules/.pnpm/[email protected]_ac1925d3b18bab26b8f85315f9db8478/node_modules/next/dist/pagesnode_modules/.pnpm/[email protected][email protected]/node_modules/styled-jsx/distThis happens because
handleExternalsperforms a check to make sure a module is resolved to the same location from both thecontextand the project directorydirbefore externalizing it:https://github.com/vercel/next.js/blob/07df897dad54fc2e46aa474707da223fa85594f0/packages/next/build/webpack-config.ts#L599
https://github.com/vercel/next.js/blob/07df897dad54fc2e46aa474707da223fa85594f0/packages/next/build/webpack-config.ts#L637
If
baseRes !== res, then the module is forced to be bundled.If we take this example:
Both of these paths actually resolve to the same location on disk:
So, I think the root cause of this issue is that the
resolveRequestfunction does not follow symlinks. If it followed symlinks, thenbaseReswould always equalres, so React would be externalized as we expect.