Ava: Reduce installed package size

Created on 26 Dec 2015  ·  19Comments  ·  Source: avajs/ava

Latest master (b79fdfd8b0e5632eae338b563af1c3be29fecbca), when installed with [email protected], is 68 MB. Most of that is Babel stuff.

We should look into ways we can reduce the size.

screen shot 2015-12-26 at 00 20 16

_(Output from DaisyDisk)_

Turns out every single Babel package includes a dependency on babel-runtime and for some reason it's not deduped by npm@3, not even with npm dedupe:

~/dev/ava/node_modules master*
❯ find . -type d -name "babel-runtime" -print 
./babel-code-frame/node_modules/babel-runtime
./babel-core/node_modules/babel-runtime
./babel-generator/node_modules/babel-runtime
./babel-helper-builder-binary-assignment-operator-visitor/node_modules/babel-runtime
./babel-helper-call-delegate/node_modules/babel-runtime
./babel-helper-define-map/node_modules/babel-runtime
./babel-helper-explode-assignable-expression/node_modules/babel-runtime
./babel-helper-function-name/node_modules/babel-runtime
./babel-helper-get-function-arity/node_modules/babel-runtime
./babel-helper-hoist-variables/node_modules/babel-runtime
./babel-helper-optimise-call-expression/node_modules/babel-runtime
./babel-helper-regex/node_modules/babel-runtime
./babel-helper-remap-async-to-generator/node_modules/babel-runtime
./babel-helper-replace-supers/node_modules/babel-runtime
./babel-helpers/node_modules/babel-runtime
./babel-messages/node_modules/babel-runtime
./babel-plugin-check-es2015-constants/node_modules/babel-runtime
./babel-plugin-syntax-async-functions/node_modules/babel-runtime
./babel-plugin-syntax-exponentiation-operator/node_modules/babel-runtime
./babel-plugin-syntax-object-rest-spread/node_modules/babel-runtime
./babel-plugin-transform-async-to-generator/node_modules/babel-runtime
./babel-plugin-transform-es2015-arrow-functions/node_modules/babel-runtime
./babel-plugin-transform-es2015-block-scoped-functions/node_modules/babel-runtime
./babel-plugin-transform-es2015-block-scoping/node_modules/babel-runtime
./babel-plugin-transform-es2015-classes/node_modules/babel-runtime
./babel-plugin-transform-es2015-computed-properties/node_modules/babel-runtime
./babel-plugin-transform-es2015-destructuring/node_modules/babel-runtime
./babel-plugin-transform-es2015-for-of/node_modules/babel-runtime
./babel-plugin-transform-es2015-function-name/node_modules/babel-runtime
./babel-plugin-transform-es2015-literals/node_modules/babel-runtime
./babel-plugin-transform-es2015-modules-commonjs/node_modules/babel-runtime
./babel-plugin-transform-es2015-object-super/node_modules/babel-runtime
./babel-plugin-transform-es2015-parameters/node_modules/babel-runtime
./babel-plugin-transform-es2015-shorthand-properties/node_modules/babel-runtime
./babel-plugin-transform-es2015-spread/node_modules/babel-runtime
./babel-plugin-transform-es2015-sticky-regex/node_modules/babel-runtime
./babel-plugin-transform-es2015-template-literals/node_modules/babel-runtime
./babel-plugin-transform-es2015-typeof-symbol/node_modules/babel-runtime
./babel-plugin-transform-es2015-unicode-regex/node_modules/babel-runtime
./babel-plugin-transform-exponentiation-operator/node_modules/babel-runtime
./babel-plugin-transform-object-rest-spread/node_modules/babel-runtime
./babel-plugin-transform-regenerator/node_modules/babel-runtime
./babel-plugin-transform-runtime/node_modules/babel-runtime
./babel-plugin-transform-strict-mode/node_modules/babel-runtime
./babel-register/node_modules/babel-runtime
./babel-runtime
./babel-template/node_modules/babel-runtime
./babel-traverse/node_modules/babel-runtime
./babel-types/node_modules/babel-runtime
./babylon/node_modules/babel-runtime
enhancement good for beginner help wanted low priority
Source
picture sindresorhus
👍7

All 19 comments

@sindresorhus: That appears to be a bug on your machine. On mine it works fine:
Screenshot
ls spits out the size kind of weird (in raw bytes, floored), it's actually 33.2MB:
Screenshot
npm ls:

/Users/Ari/Developer/.tmp
└─┬ [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ └─┬ [email protected]
  │ │   ├─┬ [email protected]
  │ │   │ ├─┬ [email protected]
  │ │   │ │ └─┬ [email protected]
  │ │   │ │   └── [email protected]
  │ │   │ └─┬ [email protected]
  │ │   │   └── [email protected]
  │ │   └── [email protected]
  │ ├── [email protected]
  │ └─┬ [email protected]
  │   └─┬ [email protected]
  │     └─┬ [email protected]
  │       └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └─┬ [email protected]
  │ │   ├── [email protected]
  │ │   └── [email protected]
  │ ├─┬ [email protected]
  │ │ └─┬ [email protected]
  │ │   └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├─┬ [email protected]
  │ │ │ │ └── [email protected]
  │ │ │ ├─┬ [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ └── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├─┬ [email protected]
  │ │ │ │ └── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └─┬ [email protected]
  │ │ │   ├── [email protected]
  │ │ │   ├─┬ [email protected]
  │ │ │   │ ├─┬ [email protected]
  │ │ │   │ │ ├─┬ [email protected]
  │ │ │   │ │ │ ├─┬ [email protected]
  │ │ │   │ │ │ │ └── [email protected]
  │ │ │   │ │ │ └── [email protected]
  │ │ │   │ │ └── [email protected]
  │ │ │   │ └── [email protected]
  │ │ │   ├── [email protected]
  │ │ │   ├─┬ [email protected]
  │ │ │   │ └─┬ [email protected]
  │ │ │   │   └── [email protected]
  │ │ │   ├── [email protected]
  │ │ │   └── [email protected]
  │ │ ├── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├── [email protected]
  │ │ ├── [email protected]
  │ │ └─┬ [email protected]
  │ │   └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├─┬ [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ ├── [email protected]
  │ │ │ │ └── [email protected]
  │ │ │ └── [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ └─┬ [email protected]
  │   └─┬ [email protected]
  │     ├── [email protected]
  │     └── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ └─┬ [email protected]
  │   ├─┬ [email protected]
  │   │ └── [email protected]
  │   ├── [email protected]
  │   └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ ├── [email protected]
  │ │ └─┬ [email protected]
  │ │   ├─┬ [email protected]
  │ │   │ └── [email protected]
  │ │   └─┬ [email protected]
  │ │     └── [email protected]
  │ ├─┬ [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ └── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └── [email protected]
  ├── [email protected]
  ├── [email protected]
  ├─┬ [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├─┬ [email protected]
  │ ├── [email protected]
  │ └─┬ [email protected]
  │   ├── [email protected]
  │   └── [email protected]
  ├─┬ [email protected]
  │ ├─┬ [email protected]
  │ │ ├── [email protected]
  │ │ ├── [email protected]
  │ │ └── [email protected]
  │ ├── [email protected]
  │ ├─┬ [email protected]
  │ │ └── [email protected]
  │ └── [email protected]
  └─┬ [email protected]
    ├─┬ [email protected]
    │ ├── [email protected]
    │ ├─┬ [email protected]
    │ │ └── [email protected]
    │ ├── [email protected]
    │ ├─┬ [email protected]
    │ │ ├── [email protected]
    │ │ └── [email protected]
    │ └── [email protected]
    ├── [email protected]
    ├─┬ [email protected]
    │ └─┬ [email protected]
    │   ├─┬ [email protected]
    │   │ ├─┬ [email protected]
    │   │ │ ├── [email protected]
    │   │ │ └─┬ [email protected]
    │   │ │   ├── [email protected]
    │   │ │   ├── [email protected]
    │   │ │   ├── [email protected]
    │   │ │   ├── [email protected]
    │   │ │   └── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├── [email protected]
    │   │ ├─┬ [email protected]
    │   │ │ └─┬ [email protected]
    │   │ │   └── [email protected]
    │   │ └── [email protected]
    │   └─┬ [email protected]
    │     └─┬ [email protected]
    │       ├── [email protected]
    │       ├── [email protected]
    │       ├── [email protected]
    │       └── [email protected]
    ├── [email protected]
    └── [email protected]

So I'm really not sure why you're seeing that. Maybe file a bug report with npm?

ariporad picture ariporad on 26 Dec 2015

Actually, it is only ~16.5 MB, @ariporad's number includes dev-dependencies. You need to delete node_modules and run

$ npm install --production

If you create a new project and just do this:

$ npm init
# default for every prompt
$ npm i babel-core babel-preset-es2015 babel-preset-stage-2

I get a relatively svelte 6.8 MB.

Adding babel-plugin-espower brings me to 9.8 MB. That seems like a pretty significant jump. @twada - Any ideas how we can reduce this?

Adding [email protected] does bump me up to 14 MB, but that is due to massive duplication of [email protected]. It gets copied into the node_modules folder of every babel-plugin-transform-*. This is expected, since current babel plugins are written in ES6, compiled with babel@5, and maintain a dev-dependency on babel-runtime@5. This really does not hurt your install times though, since all those duplicates are definitely coming from the cache (babel-runtime is not re-downloaded 20 times).

2.5 MB for the remaining 30 some dependencies we have seems pretty reasonable.

jamestalmage picture jamestalmage on 26 Dec 2015

@jamestalmage: that's strange, I did go above and beyond my usual intelligence level and create a new project to install AVA, but I'll try again when I'm home.

ariporad picture ariporad on 26 Dec 2015

What are we hoping to achieve here? With changing dependencies this is a never ending struggle.

That said, looks like all Babel dependencies require babel-runtime@^5, whereas we explicitly install ^6. There's FIFTY copies of [email protected].

Presumably we could downgrade to ^5 for the time being and we'd save a lot space. In my testing that takes us from 77MB to 29MB.

Of course there's other dependencies doing weird stuff, like fsevents which pulls in 7MB just for node-pre-gyp.

To measure I did a fresh install from NPM (in an empty directory) and ran du -k -d 1 node_modules | sort -n -r. Also check out snyk-resolve which shows the actual dependencies, not deduped.

novemberborn picture novemberborn on 29 Feb 2016

@novemberborn There are some things that can be done even if the worst offender, Babel and fsevents, are out of our control. Many packages include junk like tests or examples directory, etc. I did a PR marathon a few years ago for Yeoman on 100+ different packages adding a "files" property to their package.json to reduce the total disk size of the dependency tree. That helped a lot. Pretty much have this still open for when I have time to do the same here.

sindresorhus picture sindresorhus on 29 Feb 2016

Pretty much have this still open for when I have time to do the same here.

Sure, fair enough. Somebody should write a tool for that :stuck_out_tongue_winking_eye:

Also thinking about it more I'm doubtful we can change our babel-runtime version. Presumably the runtime transformer plugin builds against v6, not v5.

novemberborn picture novemberborn on 29 Feb 2016

Also thinking about it more I'm doubtful we can change our babel-runtime version. Presumably the runtime transformer plugin builds against v6, not v5.

Actually if we have a dependency which merely reexports babel-runtime it would allow npm to dedupe babel-runtime@5 instead.

novemberborn picture novemberborn on 1 Mar 2016

Actually if we have a dependency which merely reexports babel-runtime it would allow npm to dedupe babel-runtime@5 instead.

I thought about that. Actually, I think it would be better if Babel did that (all the Babel 6 plugins should depend on a differently named module that exports babel-runtime@5). That way everyone benefits.

jamestalmage picture jamestalmage on 11 Mar 2016

Actually, I think it would be better if Babel did that (all the Babel 6 plugins should depend on a differently named module that exports babel-runtime@5). That way everyone benefits.

:+1:

sindresorhus picture sindresorhus on 15 Mar 2016

Is detecting Node v6 possible or an option? Would we still want babel to be part of ava when the environment is ES2015-compliant?

jokeyrhyme picture jokeyrhyme on 27 Apr 2016

@jokeyrhyme AVA still injects features that have not yet landed in ES2015 or ES2016, such as async/await. Additionally, it uses Babel for internal transforms (enhancing assertions with power-assert for instance).

I don't know if it's possible to correctly detect the Node version, but that'd be at runtime, not at install time. The team could remove the ES2015 preset when all supported Node versions support it, but that could take a while.

jfmengels picture jfmengels on 27 Apr 2016
👍1

@jfmengels we might be able to install babel in a pre/post-install npm hook, and only install exactly the transforms needed to get us from the current version of node to stage-2 or whatever our threshold is. That would mean fewer dependencies for newer Nodes?

jokeyrhyme picture jokeyrhyme on 27 Apr 2016

@jokeyrhyme That's unfortunately not an option for so many reasons. Some of the best features of AVA require Babel, so I don't see why anyone would want to drop it for some minor space savings. The biggest Babel size issue is about to get fixed anyways https://github.com/babel/babel/pull/3438.

If you want to help out with this, I'd suggest you find packages in the dependency tree that includes junk like test/examples/etc folders and ask them (or better yet do a PR) to use the files property in package.json.

sindresorhus picture sindresorhus on 27 Apr 2016

Here's a list of packages we depend on that doesn't have a files entry or .npmignore. Please help us reduce the size of AVA by submitting PRs to those packages adding a files property in package.json.

Tip: $ npm i -g npm-home and then just write nh package-name to go to a specific package.

The list was generated with:

❯ package-config-checker | grep ✖ | sed 's/^ *✖/- [ ]/g'
sindresorhus picture sindresorhus on 4 May 2016

It may also be worth thinking about install _time_ as well as size. See #841 -- on my machine, AVA's install took ~50.1s. I wonder how much of that is I/O (in which case it will be fixed by addressing the size issue) vs. dependency resolution (in which case the deep nesting itself is also a cause) vs. network traffic (a bit of both, since package requests are made individually).

Another option is to pull out some of the larger deps that only cater to certain use cases into optional plugins, although I suspect the core team would prefer to avoid additional config.

billyjanitsch picture billyjanitsch on 18 May 2016
👍1

The real issue here is that npm is just slow at this. npm install ava creates hundreds of http requests that just get 3XX responses. It can't even do much of it in parallel, because it needs to resolve which version of direct dependencies will be used, before it can start querying one level deep for transitive dependencies.

I don't see us dropping lots of dependencies. The only dependencies that would be easy to replace, are also ones that are not really going to cut down on download times.

Also, I really don't think comparing cleared cache download times is that valuable. I'm much more concerned with download times when we have mostly cache hits. (AVA is still pretty bad by that metric as well, but it's not as extreme).

jamestalmage picture jamestalmage on 18 May 2016

Hello. I have been checking the list of packages, and found out that many of them already have a closed PR, specially from feross and substack authors.

I submit an updated list here with some packages edited in bold, the ones with a new PR and the ones that have an .npmignore. I also wrote the state of the related PR for each one.

The reason and previous discussions on why the PRs were not accepted can be found in feross/is-buffer#12

PACKAGE LIST

achecopar picture achecopar on 23 May 2016

https://github.com/siddharthkp/cost-of-modules is also a nice tool:

$ cost-of-modules --no-install --less

Calculating...


┌────────────────────────────────┬──────────────┬────────┐
│ name                           │ children     │ size   │
├────────────────────────────────┼──────────────┼────────┤
│ babel-preset-stage-2           │ 57           │ 23.66M │
├────────────────────────────────┼──────────────┼────────┤
│ babel-core                     │ 35           │ 15.96M │
├────────────────────────────────┼──────────────┼────────┤
│ babel-preset-es2015-node4      │ 33           │ 11.36M │
├────────────────────────────────┼──────────────┼────────┤
│ chokidar                       │ 130          │ 5.44M  │
├────────────────────────────────┼──────────────┼────────┤
│ babel-runtime                  │ 2            │ 3.82M  │
├────────────────────────────────┼──────────────┼────────┤
│ babel-plugin-espower           │ 9            │ 2.33M  │
├────────────────────────────────┼──────────────┼────────┤
│ babel-plugin-ava-throws-helper │ 6            │ 1.18M  │
├────────────────────────────────┼──────────────┼────────┤
│ jest-snapshot                  │ 10           │ 0.87M  │
├────────────────────────────────┼──────────────┼────────┤
│ bluebird                       │ 0            │ 0.58M  │
├────────────────────────────────┼──────────────┼────────┤
│ meow                           │ 25           │ 0.30M  │
├────────────────────────────────┼──────────────┼────────┤
│ + 66 modules                   │              │        │
├────────────────────────────────┼──────────────┼────────┤
│ 76 modules                     │ 345 children │ 44.53M │
└────────────────────────────────┴──────────────┴────────┘
novemberborn picture novemberborn on 14 Jan 2017
👍1

With npm 5.3, in a temp directory:

❯ npm init --yes
❯ npm i ava
❯ npx cost-of-modules --no-install --less
npx: installed 16 in 2.28s

Calculating...


┌───────────┬──────────────┬────────┐
│ name      │ children     │ size   │
├───────────┼──────────────┼────────┤
│ ava       │ 750          │ 46.04M │
├───────────┼──────────────┼────────┤
│ 1 modules │ 413 children │ 17.22M │
└───────────┴──────────────┴────────┘

The final size matches what Finder reports. I think that's pretty good! Plus, installation times are a lot better with npm 5.

Whilst we can always improve this, I don't think there's much use in keeping this issue open.

novemberborn picture novemberborn on 13 Aug 2017
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ivogabe picture ivogabe  ·  5Comments
gajus picture gajus  ·  5Comments
electerious picture electerious  ·  3Comments
avaly picture avaly  ·  4Comments
micaelmbagira picture micaelmbagira  ·  4Comments