Async: Package download size

[TehShrike]

The inclusion of lodash made the download size for async jump dramatically: http://pkgsize.com/async.html

This had the side effect of causing other highly-used packages to have their download sizes doubled or more:

• request
• node-gyp

In a world where people are already sensitive to module download size, an extra 5MIb seems undesirable.

I started to look at what was being used to change the dependencies over to the individual lodash packages published to npm in an attempt to decrease the download size, but async seems to depend on "private" lodash files that aren't published in their own modules.

I'm not sure what the best solution is here.

You could use rollup or some other tree-shaking bundler to inline lodash and eliminate the unused code, but that results in a higher bundle size for anyone using lodash in the browser.

lodash internal functions seem tightly integrated into this library, so I can only imagine how much work refactoring them out would be.

[megawac]

The main reason we decided to go this route was under the assumption that lodash would already be included in users node_modules/caches. The actual size of asyncjs distribution file size (what is loaded when you do require('async') or import the script in a browser) has not increased substantially with these changes only the size of the node_module space consumed on disk.

You could use rollup or some other tree-shaking bundler to inline lodash and eliminate the unused code, but that results in a higher bundle size for anyone using lodash in the browser.

we already do this, the main file points to a dist/ file that is finely tuned to reduce size (you noticed we use lodash internals, we do so to reduce size of the distribution size)

Also see #914

[mikermcneil]

Just to provide a data point here, between [email protected] and [email protected], for most folks, it's a difference of ~0.8 seconds as far as install time. Now granted, that differential is 800% of the entire install size back in [email protected], so it feels like a lot :p But it's really not that bad-- and like @megawac mentioned, in some cases, the lodash deps will be cached already (although I've noticed that, depending on your NPM version, it still gets very chatty about them regardless.)

Before updating Waterline's async dep:

∑ kit deps
✗ [email protected]  (should instead be pinned @2.0.1)     222.01 KB  0.111 seconds
  [email protected]                                    74.72 KB   0.037 seconds
  @sailshq/[email protected]                             428.03 KB  0.214 seconds
  [email protected]                                     581.24 KB  0.29 seconds
  [email protected]                                      849.14 KB  0.424 seconds
  [email protected]                                   877.02 KB  0.438 seconds
  [email protected]                           869.64 KB  0.434 seconds
  [email protected]                             913.68 KB  0.456 seconds
  [email protected]  (~0.11.2)                           3.74 MB    1.872 seconds
  [email protected]                                       1.21 MB    0.609 seconds

Altogether, dependencies weigh in at 9.77 MB
(that will take an average of 4.88 seconds on coffee shop internet)

 (Note that `devDependencies` and `optionalDependencies` were NOT included above.)

After updating to async@2:

∑ kit deps
  @sailshq/[email protected]       428.03 KB  0.214 seconds
  [email protected]              74.72 KB   0.037 seconds
  [email protected]               581.24 KB  0.29 seconds
  [email protected]                849.14 KB  0.424 seconds
  [email protected]             877.02 KB  0.438 seconds
  [email protected]     869.64 KB  0.434 seconds
  [email protected]       913.68 KB  0.456 seconds
  [email protected]                  1.81 MB    0.909 seconds
  [email protected]  (~0.11.2)     3.74 MB    1.872 seconds
  [email protected]                 1.21 MB    0.609 seconds

Altogether, dependencies weigh in at 11.37 MB
(that will take an average of 5.68 seconds on coffee shop internet)

 (Note that `devDependencies` and `optionalDependencies` were NOT included above.)

@TehShrike if initial download/install time is a big factor for your use case, you could consider doing something like what we did for Lodash 3.10.x. Granted, we only made that repo for an unrelated reason, but it does have the nice side effect of shrinking the download size back to what it was before, if not smaller. But be warned! You'll have to keep yours up to date in order to pick up changes from caolan/async. So I'd really make sure you care enough, and that this library is your install/download-time bottleneck. (For example, for Waterline, if I was optimizing install/download time, I'd work on anchor first before looking into async, since anchor is over twice as big.)

[aearly]

I've been thinking about this a bit more.

The reason we include Lodash is so you can require('async/waterfall')'. The library at-large has Lodash compiled in with Rollup, so it is not needed in most cases. The individual methods are standard CJS modules that require other CJS modules -- some of our internals, and individual Lodash methods, hence why Lodash is a dependency.

What if we used Rollup to flatten each of our individual methods down to single files? Lodash would become a devDependency, and requiring a single method in turn wouldn't require a dozen or more other files. There would be some code duplication, and Async itself would get larger, but it will probably be smaller than Async + Lodash. For people using only a few Async methods, their code would get smaller and faster, due to less CJS overhead.

It also might fix that bug where certain Lodash internals are missing for some people in some cases. #1352

What do you think?

[aearly]

@megawac any thoughts on the above?

[megawac]

That's definitely an option, though I think it kind of defeats the purpose of modularizing these files. Perhaps slightly better would be to copy only the lodash files we need into the repo and map the imports of lodash/filter for example to ../lodash/filter so that if you are consuming 2 async methods that were to import filter you aren't bundling the method twice. Similarly, several async methods rely on eachother, won't this cause significant internal package size bloat?

I think it might be a better option to start using the lodash npm modules, at least for the now distributable code. Switching to the npm modules would remove our dependence on lodash internal implementations, however it would likely also increase our distributable sizes as we'd start importing things like deep string support (perhaps webpack might help?).

For people using only a few Async methods, their code would get smaller and faster, due to less CJS overhead.

This is barely a consideration from my testing as most file system imports are a one time operation on the order of milliseconds and npm is getting better at doing it faster.

[aearly]

FWIW, I'm getting more bearish on our use of Lodash, especially since we're already relying on Lodash internals to cut down on bundle size. Our most commonly used Lodash method is noop, and all the others would be easy to write our own versions of.

[fregante]

Done!