Js-ipfs: Package size is 300 Mb

[rumkin]

• Version: 0.38.0
• Platform: Node.JS
• Subsystem:

Type: Overweight

Severity: Medium

Description: Package weight is 336 Mb

Steps to reproduce the error:

npm i ipfs
du -sh node_modules

Current IPFS package weight too much, it takes too long time to install. It hits performance on my pipeline and increases my bundle size dramatically.

[alanshaw]

@rumkin can I check you're quoting the size of your node_modules folder on disk after unzipping all the modules and building all the native deps?

FWIW a browser build of ipfs weighs in at around 600kb currently (which is still big) but is nowhere near 300mb!

Looking at the node_modules directory a lot of this seems to be the published standalone browser builds for each module.

[rumkin]

I've updated steps to reproduce and added size measure step, which is du -sh node_modules. I've also checked bundlephobia to get the size and received 1.5Mb and 465Kb minified, but this numbers are received by manipulation over package source. Not sure how do they get them.

I'm using IPFS as dependency to my electron app. It would be nice you provide instruction how to minify the size or (what's preferrable to me), provide a compiled bundle for node.js.

[daviddias]

Some notes from the twitter thread: https://twitter.com/rumkin/status/1185009479101964288

Suggestion from @hugomrdias https://twitter.com/hugomrdias/status/1185825396551499776

[mikeal]

npm install —production should skip the devDependencies, which you don’t need if you’re just using it as a depedency. That’ll drop the size significantly since aegir accounts for most of the size since it has to pull down a build of headless Chrome.

To test this, go into your project and

rm -rf node_modules
npm install —production

[achingbrain]

npm i in the ipfs/js-ipfs repo gives me a node_modules folder that's over 950MB(!).

npm i --production (or npm install ipfs which is what @rumkin is doing) gives me a node_modules that's 326M.

Of this 326MB, roughly:

• 197MB is folders named dist - e.g browser bundles, minified js & source maps
• 20MB is folders named docs or doc
• 12MB is in folders named test or tests
• 8MB is json - e.g. package.json, etc
• 3.5MB is compiled native code
• 3.5MB is typescript source

[mikeal]

electron app

If you’re concerned about the size of the app you may want to use https://github.com/electron-userland/electron-webpack

If you compile the entire app down with webpack only the code you rely will be there and you can do an electron build without node_modules.

[rumkin]

As the first step I'd suggest to split the package into several:

• IPFS core: for build IPFS into another products.
• IPFS node: for running standalone node as a micro/web-service as a daemon. This is what ipfs package should be, as I think.
• IPFS UI: for node management and inspection. This package could consist of two others: manager and viewer.

In my project I don't need web interface or CLI. The only thing I need is IPFS networking and storage capabilities provided by the core package.

[achingbrain]

I'm not 100% convinced this would solve the size problem.

Just under half of the LOC in /src is CLI/web related and removing them would save about 200KB (unminified & uncompressed) and a few megs of supporting dependencies which granted is not insignificant.

The problem is there's almost 200MB of browser bundles in node_modules which is what the original objection was about and almost all of those are in libp2p-*/ipld-*/ipfs-* modules that IPFS requires to function.

[rumkin]

I don't see how it conflicts with what I propose. CLI and Web pull a bunch of own dependencies like @hapi/hapi, yargs which has their own dependencies and so. Separation would make ipfs package less heavy and faster to install.

This should be moved into separate issue. But I think it should be done anyway. Currently the package is overloaded with responsibilities and tightly coupled.

[alanhoff]

on osx npm i ipfs uses 969MB of storage...

[ottodevs]

npm install —production should skip the devDependencies, which you don’t need if you’re just using it as a depedency. That’ll drop the size significantly since aegir accounts for most of the size since it has to pull down a build of headless Chrome.

To test this, go into your project and

rm -rf node_modules
npm install —production

This seems not valid since it will also skip the devDependencies of my own project...

The point is, I want to install the devDependencies declared in my package because I use them to develop my project, but I don't want to install 1gb of garbage dependencies that I will not use and that ipfs does not depend on to work, so why are we enforced to install electron?

From a technical perspective this does not seem a great design, a "consumer" of the ipfs library wouldn't need to be enforced to install 1Gb of unrelated dependencies, while the library just weights a couple hundred Kilobytes

Is there any alternative to avoid this? I suggest moving the tests to a separate package and make this package depend on aegir and ipfs so ipfs stays "dependency clean"

Another alternative seems to only publish a bundle so users can consume the pre-built ipfs dependency without needing all the extra stuff

Maybe I am missing something, if anyone knows a way to avoid installing devDependencies from ipfs, while keeping my own direct devDependencies please tell me, it makes no sense for me, I am not even depending on ipfs directly, but on 3box that uses it!!!

[hugomrdias]

@ottodevs npm does not install dev deps from your deps

what we can do is reduce the size of the dist folders.

[ottodevs]

@ottodevs npm does not install dev deps from your deps

what we can do is reduce the size of the dist folders.

Thanks for the quick answer and clarification @hugomrdias

I might be doing something wrong but when a git clone myproject then cd myproject && npm i I get this folder:
ls node_modules/ipfs/node_modules/electron along with other dependencies that are supposed to be devDependencies of my dependencies, and not declared anywhere into my package.json

I am wondering if that's a npm problem and yarn does it right, in any case, if you think reducing dist folders size will help, that would be also nice, but that electron folder shouldn't be there in my opinion when a user just adds ipfs package, as electron is not a direct dependency and is unwanted

[hugomrdias]

can you run yarn why electron and post here the output ?

[ottodevs]

Weird output...

yarn why electron
yarn why v1.22.0

[1/4] 🤔  Why do we have the module "electron"...?
[2/4] 🚚  Initialising dependency graph...
(...lots of: [warning  > 3box > ipfs > [email protected]: stable api reached] omitted)
[3/4] 🔍  Finding dependency...
error We couldn't find a match!
✨  Done in 116.89s.

... but the folder definitely exists...

ls node_modules/ipfs/node_modules/electron
LICENSE     cli.js      electron.d.ts   install.js  package.json
README.md   dist        index.js    node_modules    path.txt

Maybe is because my project does normally use npm so there is not a yarn.lock file

[hugomrdias]

rm -rf node_modules && yarn && yarn why electron

[achingbrain]

What version of ipfs do you end up with? There was a problem with the shrink wrap file for a while that mean the dev deps could sometimes get installed.

[ottodevs]

rm -rf node_modules && rm package-lock.json && yarn && yarn why electron

Still does not finds a match; after removing the node_modules and do a clean install with yarn, the electron folder is gone.

then I try again:
rm -rf node_modules && rm yarn.lock && npm i && npx npm-why electron
This shows: No one requires electron.

But in this second case back to npm, the output of ls node_modules/ipfs/node_modules/electron shows that electron is present again, so the issue seems to be with npm installing devDepencencies somehow... in this case the node_modules/ipfs folder size grows over 1GB

Maybe is time to encourage my team to shift to yarn, pnpm or similar...

[ottodevs]

What version of ipfs do you end up with? There was a problem with the shrink wrap file for a while that mean the dev deps could sometimes get installed.

npm ls ipfs
[email protected]
  └─┬ [email protected]
    └─┬ [email protected] 
      └── [email protected]  extraneous

npm ERR! extraneous: [email protected] /Users/otto/box/git/my-project/node_modules/ipfs/node_modules/ipfs

[achingbrain]

0.40.0 had the npm-shrinkwrap.json file with the extra dependencies in it - this will be fixed when 3box upgrade to 0.41.1 or later.

[achingbrain]

The situation here has changed quite a lot in the intervening months. We've pulled a bunch of uncommonly used deps out of ipfs, so right now doing npm i ipfs in an empty directory results in a node_modules folder that's 167MB in size, a significant reduction from the 336MB originally reported.

If you are only running an in-process node as part of an application and don't need the CLI or HTTP API server components you can reduce the install size by a further 30MB by using the ipfs-core module which is a drop-in replacement for ipfs in your application code.

We'll keep trying to slim the install down but as it stands pressure on CI and other space/bandwidth constrained environments should now be significantly lightened.