Docusaurus: Minify failure when building docusaurus site in a yarn workspaces with another project

[taylorreece]

Edit from @slorber: TL.DR, try using this nohoist config

🐛 Bug Report

I have a Docusaurus-built site that shares a yarn.lock file with several other projects. Since bumping to Docusaurus to 2.0.0-alpha.64 from 2.0.0-alpha.63, I have been unable to docusaurus build. I get this error:

(undefined) TypeError: Cannot read property 'replace' of undefined
    at Object.options.minifyJS (main:18614:28)
    at Object.chars (main:19102:24)
    at main:48074:19
    at String.replace (<anonymous>)
    at new HTMLParser (main:48066:19)
    at minify (main:18888:3)
    at module.exports.exports.minify (main:19249:16)
    at serverEntry_render (main:87861:38)
Error: Failed to compile with errors.
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/@docusaurus/core/lib/webpack/utils.js:164:24
    at finalCallback (/Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/MultiCompiler.js:254:12)
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/MultiCompiler.js:277:6
    at done (/Users/treece/src/docusaurus-minify-troubles/node_modules/neo-async/async.js:2931:13)
    at runCompilers (/Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/MultiCompiler.js:181:48)
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/MultiCompiler.js:188:7
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/MultiCompiler.js:270:7
    at finalCallback (/Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/Compiler.js:257:39)
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/Compiler.js:273:13
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/treece/src/docusaurus-minify-troubles/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:40:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/treece/src/docusaurus-minify-troubles/node_modules/tapable/lib/Hook.js:154:20)
    at onCompiled (/Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/Compiler.js:271:21)
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/Compiler.js:681:15
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/treece/src/docusaurus-minify-troubles/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:4:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/treece/src/docusaurus-minify-troubles/node_modules/tapable/lib/Hook.js:154:20)
    at /Users/treece/src/docusaurus-minify-troubles/node_modules/webpack/lib/Compiler.js:678:31
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Have you read the Contributing Guidelines on issues?

Yes

To Reproduce

I've created a minimal repro repo, https://github.com/taylorreece/docusaurus-minify-troubles, with a new Docusaurus project created with npx @docusaurus/init@next init docs classic. I did not touch the docusaurus project otherwise.
The other project in this repo, aptly named "other-project" has a package.json file with a series of dependencies.
These two projects share a yarn.lock.

1. git clone [email protected]:taylorreece/docusaurus-minify-troubles.git
2. cd docusaurus-minify-troubles
3. yarn install
4. cd docs/
5. yarn build

If you remove other-project and then yarn install and yarn build, the Docusaurus site builds fine.

Expected behavior

I'd expect it to build.

Actual Behavior

I get the stack trace listed above

Your Environment

MacOS
Node v12.16.2
Yarn v1.22.4

[taylorreece]

New info: one of my other projects has a dependency that has a dependency on "file-loader": "^1.1.11". That seems to be causing me grief.

If I have a project in my yarn workspaces with just that dependency, I can replicate this issue. I've updated my repro repo to show an extra project with just that dependency. Somehow that older version file-loader is making its way into the Docusaurus build?

For my repro repo, I can build my docs project successfully if I add an explicit dependency to "file-loader": "^6.1.0", to my docusaurus project. That doesn't seem to help for my non-repro repo sadly...

My stack trace references @docusaurus/core/lib/webpack/utils.js. It does some require.resolve()'s instead of some imports - https://github.com/facebook/docusaurus/blob/5359d61d9e6aad824b8185117ed167e0aadecd3b/packages/docusaurus/src/webpack/utils.ts#L214. I'm too dumb to understand node imports, but maybe that's yanking in the old file-loader version?

[taylorreece]

It seems to not be an issue with just file-loader, but with webpack looking into other projects' node_modules directories in general for dependencies. Maybe the webpack config that Docusaurus uses to build is configured to look in all projects, rather than in the local project and root directory (in that order)?

[taylorreece]

Temporary workaround: if you nohoist the docusaurus project, and remove all node_modules directories and run yarn install again, the problem goes away. You do get duplicated directories in dependencies, as all of the docusaurus project's dependencies are no longer shared, but that's better than builds failing 🤷

{
  "name": "my-shared-project",
  "private": true,
  "workspaces": {
    "packages": [
      "docs",
      "other-project"
    ],
    "nohoist": [
      "docs/**"
    ]
  }
}

[slorber]

Hi,

It seems it's not the webpack terserPlugin (that you can disable with the cli option) that is causing this problem, but rather the HTML minifier (not possible to disable atm).

https://github.com/terser/html-minifier-terser

It's worth checking with yarn why which version is actually used. We run [email protected] but maybe another version has a bug or something?

You could modify the "serverEntry.js" file locally to check what's happening and see if removing the minifyJS: true solves it

[craigpalermo]

We're also experiencing a similar issue with docusaurus build. Development mode works fine, but the server has compilation errors and docusaurus build results in the following stack trace. We think it began in 2.0.0-alpha.63, but can't tell for sure since we can't get back to a working state. Our repo uses Rush and PNPM, so the docs project shares a lockfile with other projects. (Edit: I was able to repro when installing via NPM outside of our monorepo).

Error: Cannot find module './137.2ef14012.js'
Require stack:
- main
    at Array.map (<anonymous>)
(undefined) Error: Cannot find module './137.2ef14012.js'
Require stack:
- main
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:15)
    at Function.Module._load (internal/modules/cjs/loader.js:842:27)
    at requireLike (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/require-like/lib/require-like.js:15:26)
    at Function.requireEnsure [as e] (main:46:25)
    at loader (main:50907:36)
    at load (main:4973:17)
    at init (main:5066:13)
    at Function.preload (main:5108:14)
    at main:51523:24
    at Array.map (<anonymous>)
    at preload (main:51518:30)
    at _callee$ (main:52003:20)
    at tryCatch (main:33082:40)
    at Generator.invoke [as _invoke] (main:33312:22)
    at Generator.next (main:33137:21)
    at asyncGeneratorStep (main:49240:24)
Error: Failed to compile with errors.
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/@docusaurus/[email protected]_381da2c75b6848726c5f82ebd358cbf9/node_modules/@docusaurus/core/lib/webpack/utils.js:164:24
    at finalCallback (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/MultiCompiler.js:254:12)
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/MultiCompiler.js:277:6
    at done (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/neo-async/async.js:2931:13)
    at runCompilers (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/MultiCompiler.js:181:48)
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/MultiCompiler.js:188:7
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/MultiCompiler.js:270:7
    at finalCallback (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/Compiler.js:257:39)
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/Compiler.js:273:13
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:40:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/tapable/lib/Hook.js:154:20)
    at onCompiled (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/Compiler.js:271:21)
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/Compiler.js:681:15
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:4:1)
    at AsyncSeriesHook.lazyCompileHook (/Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]/node_modules/tapable/lib/Hook.js:154:20)
    at /Users/cpalermo/priceline-web/pcln-web/common/temp/node_modules/.pnpm/[email protected]_93ca2875a658e9d1552850624e6b91c7/node_modules/webpack/lib/Compiler.js:678:31

Update 2:
For some inexplicable reason, this change resolved the issue in our build:

// Before
 module.exports = { presets: ['react-app'] }

// After
module.exports = {
  presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
}

[slorber]

Thanks @craigpalermo

If you have a babel config at the root of your site, it should rather have the docusaurus preset indeed. Why do you need the react-app preset in the site folder?

Without access to the source code it's hard to see what could go wrong.

[craigpalermo]

@slorber I'm not entirely sure why we used the react-app preset in the first place.. I actually got the idea to switch over to the Docusaurus preset by comparing our project against a newly created Docusaurus project.

For future reference, do you know if there's a better way to get visibility into the build process? I couldn't find anything like a verbose flag for docusaurus build in the docs. I think it was mostly by chance that I found that it was a Babel issue 😬

[slorber]

I don't really know a good way to debug webpack builds unfortunately. BTW your issue does not really seem related to the one of @taylorreece and seems fixed, so I'll hide the comments. If you want to discuss more about your problem, let's continue on discussions (or Discord)

[cmdcolin]

I think I ran into the same issue. I get yarn why html-minifier-terser to be 5.1.1, with react-scripts contributing.

Poking into the node_modules/@docusaurus/core/lib/client/serverEntry.js and removing minifyJS: true line did fix the build

[cmdcolin]

Note I have a monorepo setup also...I had considered taking the website out of the monorepo but curious if anything comes from this one

[taylorreece]

Interesting. I'm also pinned to html-minifier-terser 5.1.1. Though, if I replace my yarn lock file with 5.0.5, I still see these issues.

[lunelson]

This is a pretty bad bug. I'm surprised the Docusaurus team has not run in to it, since they are also using a yarn workspace structure 🤔. I am also working in a monorepo, with a custom theme under /packages/ and my test site under /sites/, and the test site wouldn't build after bumping up from alpha.63. I finally managed to get a nohoist config that seems to work but this feels like a very hacky solution, that only happens to work right now because html-minifier-terser is a first-level dependency

{
  "workspaces": {
    "packages": [
      "packages/*",
      "sites/*"
    ],
    "nohoist": [
      "*/@docusaurus/*"
    ]
  }
}

[slorber]

Hi,

Tried to hack a bit the repro from @taylorreece and found the issue:

In Docusaurus core we run this static html files minification, which includes JS minification using the terser package.

This html minify package calls terser:

The problem is that terser.minify now returns a promise, and there's no "await", so this fails:
https://github.com/terser/terser/commit/f390a429b0eacd3542e2fc64bdd5ae7b16d3f180

---

I'm not sure to understand why adding/removing the file-loader dependency does affect which version of terser is used by this plugin, but in practice it does, and the html minification plugin ends up running an incompatible, newer version of terser, while it should stick to v4.

We can see 2 different outputs according to the presence or not of the file-loader dependency in the other package:

Not sure how it happens, but despite the "terser": "^4.6.3" deps, html minifier ends up running terser 5.x

[slorber]

Related to this issue: https://github.com/terser/html-minifier-terser/issues/49

Also looks related to this Docusaurus upgrade: https://github.com/facebook/docusaurus/pull/3439/files

[slorber]

I was able to make the repro repo build with the following:

  "workspaces": {
    "packages": [
      "docs",
      "other-project"
    ],
    "nohoist": [
      "**/html-minifier-terser",
    ]
  },

Note: I'm not sure the "nohoist" config works with just yarn install, had to regenerate the whole lockfile to make it work.

Please let me know how this workaround works for you.

I'm going to close this issue because I'm not sure we can do anything about it currently, really looks related to how Yarn install dependencies. We'll upgrade html-minifier-terser as soon as it upgrades to Terser v5 so that this becomes a non-problem and we only use a single version of Terser.

[taylorreece]

Hey @slorber - thanks a bunch for looking into this!

    "nohoist": [
      "**/html-minifier-terser",
    ]

ended up working. I did _not_ need regenerate a new yarn.lock file, but instead removed all node_modules directories and re-ran yarn install.

[slorber]

Great to know 👍

Let me know if this workaround is not good enough, but I don't have anything better to propose for now ;)

[cmdcolin]

@slorber great sleuthing :) thanks so much...will hope to see the dependencies catch up

[xianshannan]

It worked for lerna config below.

{
  "packages": ["packages/*", "website"],
  "version": "1.37.4",
  "npmClient": "npm",
  "hoist": true,
  "nohoist": ["@docusaurus/*"]
}

Just set the nohost for @docusaurus/*.

[the-spyke]

My deps tree using @docusaurus/core@^2.0.0-alpha.66:

@docusaurus/core@^2.0.0-alpha.66:
    html-minifier-terser "^5.0.5"
    html-webpack-plugin "^4.0.4"
    terser-webpack-plugin "^4.1.0"
    webpack "^4.44.1"

html-minifier-terser@^5.0.1, html-minifier-terser@^5.0.5:
    terser "^4.6.3"

html-webpack-plugin@^4.0.4:
    html-minifier-terser "^5.0.1"

terser-webpack-plugin@^1.4.3:
    terser "^4.1.2"

terser-webpack-plugin@^4.1.0:
    terser "^5.3.4"

webpack@^4.44.1:
    terser-webpack-plugin "^1.4.3"

Which gets resolved by Yarn into node_modules structure:

@docusaurus
    /[email protected]
        /[email protected]
        /[email protected]
[email protected]
    <none>
[email protected]
    <none>
[email protected]
    <none>
[email protected]
    <none>
[email protected]
    <none>

As it should be.

The only user of terser@5 is Docusaurus, which is also a user of html-minifier-terser and html-webpack-plugin. Both of which use older terser@4. Looks like Docusaurus gives a config in terser@5 format to those plugins as well, which fails with an error.

Adding Yarn resolution to lower terser-webpack-plugin temporary solves the issue:

{
  "resolutions": {
    "**/terser-webpack-plugin": "^1.4.3"
  }
}

[slorber]

As it should be.

I don't agree @the-spyke , If html-minifier-terser requires Terser 4, it should rather install a duplicate Terser4 package under html-minifier-terser node modules, which is not the case. As the 2 terser versions are incompatible, we should expect 2 versions to be used in practice. It is what actually happens in non-monorepos, or when you use nohoist, and what we actually want.

Note: we upgraded Webpack terser plugin on purpose to upgrade a transitive dep having a security issue (https://github.com/facebook/docusaurus/pull/3439/files)

[the-spyke]

@slorber terser@4 is required by 2 packages, terser@5 by one. Yarn hoists the most used terser@4 to the top level, and less used terser@5 is installed into the subtree reducing node_modules size. Yarn does everything right and according to the Node resolution algorithm, while simultaneously flattening the tree. If there's an issue, then somebody is loading a module from an invalid location because of wrong require.resolve, changed working directory, custom modules path, etc.

[slorber]

@the-spyke I don't know what to say, but if a package declares needing terser 4, and yarn makes it use terser v5, it looks unexpected to me, as we really want to stick to v4. Hoisting should work as an optimisation without changing the behavior. If disabling hoisting fix the app, and enabling hoisting crashes the app, for me the problem is that the hoisting optim logic is bad, it's not supposed to break anything. When running with hoisting, we don't want Yarn to give us unpredictable versions at runtime.

I don't know what to do about this apart proposing a workaround until we upgrade everything to use terser v5 (which is not yet possible)

[the-spyke]

@slorber Yarn doesn't make it use tesrser@5. The node_modules structure in my message is valid and corresponds to the specified dependencies and Node resolution algorithm.

If the structure is valid, then it's an app issue loading wrong libraries. And there're tons of ways to do this, especially with Webpack.

[sibelius]

what is the best solution without using nohoist?

[the-spyke]

@sibelius You may try my approach with setting module resolution in Yarn:

{
  "resolutions": {
    "**/terser-webpack-plugin": "^1.4.3"
  }
}

[slorber]

@sibelius unfortunately wasn't able to find another better solution, but resolutions can probably help too.

Why is it a problem to nohoist only a few packages?

[sibelius]

I've fixed like this

resolutions: {
"terser" : "4.8.0"
}