Parcel: Duplicated modules in output file

[avchugaev]

🐛 bug report

In my TypeScript + React project I've faced the problem with duplication of module in output JavaScript bundle. Same class imported two times.
Project is monorepo maintained with Lerna.

🎛 Configuration (.babelrc, package.json, cli command)

.babelrc

{
  "presets": [
    [
      "env",
      {
        "targets": {
          "browsers": [
            "last 1 Chrome version"
          ]
        }
      }
    ]
  ]
}

package.json

{
  "private": true,
  "version": "0.9.0",
  "description": "React Portlet",
  "keywords": [],
  "scripts": {
    "serve": "parcel main/html/index.html -d build/debug",
    "build": "parcel build main/html/index.html -d build/release",
    "test": "td"
  },
  "dependencies": {
    "react": "^16.4.1",
    "react-dom": "^16.4.1",
    "@monument/core": "^0.9.0",
    "@monument/react": "^0.9.0"
  },
  "devDependencies": {
    "@types/react": "^16.4.6",
    "@types/react-dom": "^16.0.6",
    "parcel-bundler": "^1.9.7"
  }
}

CLI command:

$ npm run serve

🤔 Expected Behavior

I think that all modules should accur only once in output bundle.

😯 Current Behavior

Currenly some classes are duplicated in output bundle.

💁 Possible Solution

I think root of the problem is symlink structure created by Lerna.

🔦 Context

In project I don't use code splitting yet.
At some point of time with help of logging I've found that some classes are duplicated in output bundle.

💻 Code Sample

In my case module with class "Class" is duplicated as well as probably some others.

🌍 Your Environment

| Software | Version(s) |
| ---------------- | ---------- |
| Parcel | 1.9.7
| Node | 10.5.0
| npm | 6.1.0
| Lerna | 2.11.0
| Operating System | Windows 10, Ubuntu

[avchugaev]

Possible problem is symlinks structure created by Lerna. Output javascript bundle includes same classes but under different file paths. "Unwrapping" symlinks may help in this situation. I mean paths should be pre-processed before registering a chunk to use real file path as chunk ID.

For example all these paths points to the same file

<root>/packages/core/main/reflection/Class.js
<root>/packages/react/node_modules/@monument/core/main/reflection/Class.js

but <root>/packages/react/node_modules/@monument/core is symlink that points to <root>/packages/core not a directory itself.

[avchugaev]

@DeMoorJasper , this bug is really blocker. Will you try to fix it in nearest future? It's one of core features (resolving file paths) and it does not work properly. I can give more information about use-case if needed.

[jeanfortheweb]

Same Problem for me, module gets executed twice.

[Offirmo]

Same problem for me, a monorepo handled by bolt, my modules are all duplicated, even when they have correct matching dependencies and peerPependencies.

The worst is react-dom being present 3x time in the final bundle :-(

[codefrau]

Same problem in our (non-react) monorepo. To illustrate, let's say we have 2 modules and 1 app:

modA:
   // no imports

modB:
  import "@comp/modA"

app: 
  import "@comp/modA"
  import "@comp/modB"

The folder structure created by npm i looks like this :

root
  modA
  modB
    node_modules
      @comp
        modA -> ../../../modA
  app
    node_modules
      @comp
        modA -> ../../../modA
        modB -> ../../../modB

But Parcel appears to not resolve the symlinks, it sees this as:

root
  modA
  modB
    node_modules
      @comp
        modA
  app
    node_modules
      @comp
        modA
        modB
          node_modules
            @comp
              modA

leading to modA being duplicated in the bundle:

"@comp/modA": ...
"@comp/modB": ...
"@comp/modB/node_modules/@comp/modA": ...
"app": ...

... which is obviously undesirable.

[codefrau]

Btw, here's the workaround I'm using currently. I made a package (pasted below) which deduplicates everything that has the same source code:

// Work around Parcel package duplication bug
// https://github.com/parcel-bundler/parcel/issues/1838

// Usage: import this to deduplicate modules
// Author: Bert Freudenberg (https://github.com/bertfreudenberg)

// find duplicates of modules by comparing source code
const parcel = module.bundle;
const sources = {};
const duplicates = {};
for (const dupe of Object.keys(parcel.modules)) {
    const source = "" + parcel.modules[dupe][0];
    const id = sources[source];
    if (id) duplicates[dupe] = id;
    else sources[source] = dupe;
}
// replace references to duplicates with the actual modules
const later = {};
const fixed = new Set();
for (const m of Object.values(parcel.modules)) {
    for (const [n, dupe] of Object.entries(m[1])) {
        const id = duplicates[dupe];
        if (id) {
            if (parcel.cache[dupe]) later[id] = dupe;      // dupe already loaded
            else {
                m[1][n] = id;                              // use id
                delete parcel.modules[dupe];               // delete dupe
                fixed.add(n);
            }
        }
    }
}
for (const m of Object.values(parcel.modules)) {
    for (const [n, id] of Object.entries(m[1])) {
        const dupe = later[id];
        if (dupe) {
            m[1][n] = dupe;                                 // use dupe
            delete parcel.modules[id];                      // delete id
            fixed.add(n);
        }
    }
}
for (const fix of [...fixed].sort()) console.log("Deduplicated import of", fix);

[cruzeiro99]

I'm using code splitting with react.lazy, and in routes as well, I made a workaround for this in my project which would be more easier in my case, I imported the icons that I had problems(just those 2) and put them together in a global file

//utils.js
import TopArrow from "Icons/general/whiteTopArrow.svg"
import DownArrow from "Icons/general/whiteDownArrow.svg"

window.icons = {
    ...window.icons,
    TopArrow,
    DownArrow
}

//index.js
import "utils.js"



md5-b62270a7ace560f446c44b206049915e



//component.js
export default () => <img src={window.icons.TopArrow}/>

Hope it helps

[dharmax]

I'm experiencing the same issue. It appears out of nowhere, after i did some unrelated changes.
It's with a js file. my fe router. Naturally, it kills my app and i still didn't find a working workaround.

[lrgalego]

Btw, here's the workaround I'm using currently. I made a package (pasted below) which deduplicates everything that has the same source code:

// Work around Parcel package duplication bug
// https://github.com/parcel-bundler/parcel/issues/1838

// Usage: import this to deduplicate modules
// Author: Bert Freudenberg (https://github.com/bertfreudenberg)

// find duplicates of modules by comparing source code
const parcel = module.bundle;
const sources = {};
const duplicates = {};
for (const dupe of Object.keys(parcel.modules)) {
    const source = "" + parcel.modules[dupe][0];
    const id = sources[source];
    if (id) duplicates[dupe] = id;
    else sources[source] = dupe;
}
// replace references to duplicates with the actual modules
const later = {};
const fixed = new Set();
for (const m of Object.values(parcel.modules)) {
    for (const [n, dupe] of Object.entries(m[1])) {
        const id = duplicates[dupe];
        if (id) {
            if (parcel.cache[dupe]) later[id] = dupe;      // dupe already loaded
            else {
                m[1][n] = id;                              // use id
                delete parcel.modules[dupe];               // delete dupe
                fixed.add(n);
            }
        }
    }
}
for (const m of Object.values(parcel.modules)) {
    for (const [n, id] of Object.entries(m[1])) {
        const dupe = later[id];
        if (dupe) {
            m[1][n] = dupe;                                 // use dupe
            delete parcel.modules[id];                      // delete id
            fixed.add(n);
        }
    }
}
for (const fix of [...fixed].sort()) console.log("Deduplicated import of", fix);

@bertfreudenberg How do you use this package on your parcel?

[codefrau]

@bertfreudenberg How do you use this package on your parcel?

Oh, you simply include it in your app as one of the first things to be imported/executed.

It's not a real fix, just a hacky work-around that works for us: parcel will still create a bundle with duplicate modules, but then when running the app my code goes through all modules replacing the duplicates. For this to work, my code needs to be executed before the duplicated modules (or at least before importing the duplicate).

[lrgalego]

That makes sense. I'm looking into a way to update the bundle before it packs, to apply the same logic you did on this package, but I haven't found a way (without forking parcel) to do this.
Let me know if you have any ideas on how to do it

[skipjack]

Seeing the same issue when npm link-ing a package where react is also installed. With webpack, you can customize resolution via the following configuration to fix this issue:

resolve: {
    modules: [ path.resolve(__dirname, 'node_modules'), 'node_modules' ]
}

Which essentially instructs webpack to always resolve packages in the main project's node_modules directory prior to looking in any other node_modules directory (e.g. the one in the linked package). I looked here but wasn't able to find any analogous configuration option for parcel.

This is really the only major blocker I've run into with parcel, otherwise it has been great and made projects much easier to get off the ground. IMO the behavior I described above is a pretty sensible default that would address the issue for most users (in the spirit of sensible defaults and little config). I'd be happy to submit a PR for this if the maintainers agreed.

[skipjack]

It seems there is a workaround for this with alias:

__package.json__

  "alias": {
    "react": "./node_modules/react"
  }

The downside of this is that (I think) you would need to explicitly enter a line for each duplicated package. Webpack has a similar alias option that can be used the same way though the modules approach is a bit cleaner.

[lrgalego]

@skipjack I created a repo to test the alias workaround you mentioned. but I got an interesting result.
When I run the dev server, the dependencies are duplicated in the bundle. (adding the alias fix the issue).
But when I run parcel build the dependencies are not added to the bundle. With our without the alias.
Here is my sample repo: https://github.com/lrgalego/parcel-test
That makes me believe that the code that trims the duplicated package is already in place, but it's not being invoked for the dev server.
I tested this with parcel 1.12.4

[jeanfortheweb]

i don't understand. i fixed this one years ago, including a pr with additional test cases. why is this still an issue?

[lrgalego]

@jeanfortheweb I sent a repo with a simple app showing the issue. maybe there's some edge case not covered by the test cases.
https://github.com/lrgalego/parcel-test

[lrgalego]

@bertfreudenberg @skipjack
Ok, I tried something different and it works like a charm.
Instead of making all the packages being linked to each other, I used yarn workspaces: https://classic.yarnpkg.com/en/docs/workspaces/

this way all the common dependencies are in the same directory and yarn symlink everything
so pkg1 looks for the dep on ../node_modules and pk2 also looks there. the path is the same and the dep is added only once in the bundle.
I created a new branch on my repo with the version that works well: https://github.com/lrgalego/parcel-test/tree/works-with-yarn-workspaces

[Offirmo]

Could that be added in parcel 2 roadmap?

[Offirmo]

@lrgalego are you sure your setup is complex enough?

The problem arise due to this:

• app
  
  
  
  • node_modules/modA
  
  
  • node_modules/modB
  
  
  • node_modules/modB/node_modules/modA (symlinked to modA above)
  
  

Due to require/import finding the closest matching module and then parcel seeing different pathes, modA could end up being duplicated even though it's the same module. Yarn workspaces doesn't help there.

• If the app requires modA, it is picked from app/node_modules/modA
• if modB requires modA, it is picked from app/node_modules/modB/node_modules/modA

[lrgalego]

What I have: all packages I wrote in a monorepo referencing each other by links. That's why .moving to yarn workspace, where the path is the same for all my packages solved the issue

[Offirmo]

@lrgalego I'm using workspaces as well, that's not the point. The modules are linked but since they have their own underlying node_modules, alternate pathes get picked. I edited my example above to make it clearer.

[danieltroger]

Hey guys, I'm also having the problem that parcel is including a file twice in the outputted bundle. I tried creating a module with just the file and adding it as dependency in the package.json file in the hopes that it only gets included once, but it's still twice in the output file.

And it sucks, because it adds 140KB to my final bundle :(

[danieltroger]

Just stumbled upon this duplication issue again, as seen in --detailed-report:

├── node_modules/lib/library.ts                                          7.72 KB    704ms
├── ../anothermoduleimade/node_modules/lib/library.ts                    7.72 KB    231ms

IMO this really should be fixed somehow that makes it impossible that the same file gets included twice, i.e. by comparing the filesize of every file with the file size of every file and if one matches calculating the hash of the contents and if the hash matches only including the file once.

Edit: and yeah lib is a symlink which parcel doesn't understand:

node_modules$ readlink lib
../../lib

[mischnic]

by comparing the filesize of every file with the file size of every file and if one matches calculating the hash of the contents and if the hash matches only including the file once.

This would break some react libraries which have multiple files containing import React from "react"; export default React.createContext(); and rely on them being not identical (===). https://github.com/parcel-bundler/parcel/issues/3523

Does this still happen with Parcel 2?

[danieltroger]

@mischnic thanks for the reply. I don't know how to update to v2, is there a guide?

This would break some react libraries which have multiple files containing import React from "react"; export default React.createContext(); and rely on them being not identical (===). #3523

That's actually funny because I rely on the behaviour of the modules being identical when imported from multiple places. So when I export an object I can modify it from one place I imported the module and it changes for all. Maybe it could be made configurable?

AFAIK if you need to clone an object you can return it from a function and call the function every time you want a copy.

So maybe my suggested deduplication paired with a configurable wrapping inside a function would work?

It could either be an option to the import statement but that's probably illegal or inside some config file or CLI option.

So you deduplicate everything for filesize reasons and then if people want to actually include two copies of the same file you just give them new copies in JS instead of sending 2.

[mischnic]

Rule of thumb: try it with Node's CJS behaviour or Node/browsers native ESM. This is how Parcel should behave, we are not trying to invent our own module format

[mischnic]

is there a guide?

Using parcel@nightly instead of parcel-bundler.
https://v2.parceljs.org/getting-started/migration/

new copies in JS instead of sending 2.

Something I forgot: if you have two identical files with relative imports, then the these will point to different files. They wouldn't if you deduplicate by file contents.