Mocha: Possibly speed up dev-dependency installs?

[ScottFreeCode]

I've probably been watching too many Travis builds lately, but it seems like at least half (and sometimes most) of the two to five minutes taken by non-browser jobs is just installing through NPM. With several times as many jobs in our build matrix as Travis allows us to run at once, this can really slow each build down a lot. Mocha installs quickly enough as a dependency of other projects, it's all those dev dependencies that make its own CI installs (and local development installs) sloooow.

Caching may or may not be a viable strategy to speed any of that up -- obviously we want to reinstall cleanly if anything that could affect it changes, and obviously running on so many different environments there isn't going to be a whole lot of opportunity to share an install between multiple jobs even in the same build. I suppose all the Node 8 environments could be sped up by sharing an NPM cache (not the install itself but the cache NPM uses to install from), but that still leaves the variety of non-browser Node test jobs...

However, not all of our dev dependencies are actually needed by all jobs. Could we perhaps make them optional dev dependencies and install them where they're needed? Is optional dev dependencies a thing?

Here's what I believe are prematurely pessimizing our installs by being downloaded in all of them:

• watchify appears to be referenced nowhere in our codebase except package.json; unless I've missed something, we can just remove it
• eslint and eslint-* are only used in the lint job to my knowledge
• the following are only used in browser jobs, to my knowledge:
  
  
  
  • browserify
  
  
  • browserify-istanbul (in my browser coverage branch/PR)
  
  
  • karma
  
  
  • karma-*
  
  
  • os-name? the one place I know it's being used is to determine whether to use Phantom or Chrome in a non-Sauce browser test
  
  
  • ~rimraf?~ (nope, that's actually being used in a node-only test, and if we want to get off the Makefile we'll probably need rimraf to clean the build before publishing as well as before at least some testing)
  
  
  • through2
  
  
• the following are only used in the PhantomJS browser job, to my knowledge:
  
  
  
  • phantomjs
  
  
  • karma-phantomjs-launcher
  
  
• karma-sauce-launcher is only used in Sauce browser jobs, to my knowledge
• karma-chrome-launcher is used if somebody runs the browser tests locally on Mac, but not in CI as far as I can tell
• the following are used in CI, extensively with my coverage enhancements, but might not be needed locally unless you want to generate coverage reports:
  
  
  
  • coveralls
  
  
  • istanbul-combine
  
  
  • nyc
  
  

Obviously, whatever makes these get pulled in for the relevant CI would also need to happen for local uses of them in order for us to be able to not download them by default...

And if there's no optional dev dependencies, then removing them from package.json may be dubious anyway (they'd be depended upon but untracked...).

Anywho, I figured I'd throw the idea out there and see if anybody wants to look into it further. It would be awesome to bring the install time down, both locally and to speed up CI builds; but it would depend on really being able to get this all correct...

[j-f1]

You could make package-<task>.json files and have a script rename the appropriate one to package.json.

[kunagpal]

@ScottFreeCode As far as caching is concerned, both Travis and AppVeyor are smart enough to disregard the cache when dependencies have changed. Travis does it by default, AppVeyor settings can be tweaked as follows:

cache:
  - node_modules -> package.json

[ScottFreeCode]

@kunagpal is this behavior by any chance version-range aware? It would be really awesome if we'd reinstall in situations like:

• mocha depends on X version ^1.0.0, X releases 1.1.0
• mocha depends on X version 1.0.0, [email protected] depends on Y version ^2.0.0, Y releases 2.1.0

It'd be even cooler if it would also not reinstall if we change a field in package.json that does not affect dependencies.

AppVeyor might not be as crucial as there are fewer jobs in its matrix to begin with, for what it's worth.

[kunagpal]

@ScottFreeCode To add to the point made earlier, the package.json specification in appveyor.yml is only limited to the family of dependencies. So, bumping the Mocha version, changing the License, et cetera, would not invalidate the npm cache. I haven't really experimented with what happens in the case of non-strict dependency versions. One possible way here would be to use precise dependency versions, but I feel that has been avoided with Mocha for the reasons of convenience(?).

[ScottFreeCode]

To add to the point made earlier, the package.json specification in appveyor.yml is only limited to the family of dependencies. So, bumping the Mocha version, changing the License, et cetera, would not invalidate the nom cache.

:+1:

I haven't really experimented with what happens in the case of non-strict dependency versions. One possible way here would be to use precise dependency versions, but I feel that has been avoided with Mocha for the reasons of convenience(?).

I'm actually not sure whether we're using dependency version ranges on purpose or not, but I think it's worth noting that we'd need some kind of shrinkwrap or lock to deal with version ranges in dependencies' dependencies even if Mocha were using precise versions... unless the same functionality that ignores non-dependency changes to package.json also looks at the resulting dependencies that would actually be installed, including dependencies' dependencies, rather than just at what's specified in our file(s).

(I'm also not certain what all the pros and cons are of different possible behaviors with regard to dependency versions. Might need some discussion with the team. I can imagine outdated dependencies in CI causing us problems, but I can also imagine dependencies changing in CI as third-party updates occur potentially making it hard to track down problems too.)

[kunagpal]

In my opinion, stale dependencies bring with them a greater baggage as opposed to newer dependencies. Since Mocha already uses Greenkeeper (albeit suboptimally), failures related to changes can be tracked down with relative ease.

This is slightly off topic, but including a shrinkwrap is a good idea, and will also require taking a look at https://github.com/greenkeeperio/greenkeeper-lockfile.

[aledalgrande]

Different approach suggestion, did you test how long it takes to install the deps with yarn instead of npm?

[ScottFreeCode]

There were some attempts to try switching to Yarn a while back, but I don't know if anyone timed it.

Is my understanding correct that most of the performance gains boasted by Yarn and by NPM 5 require saving a (system level) cache?

We also have in the past needed to ensure that we test installing Mocha's production dependencies with the version of NPM that is bundled with the supported versions of Node, but I don't know for sure if that's still necessary. (I believe it used to ensure we didn't include dependencies that used ^ in their version ranges... or was it ~?... but I'm fairly sure we dropped support for the version of Node that had that limitation in 3.x, and don't know if there are any remaining potential backwards/forwards compatibility issues with NPM.) Just something we'd want to double-check before relying on a newer package manager (whether Yarn or NPM 5).

[aledalgrande]

Unfortunately at this point I don't know much about this, but have seen the difference on a few projects and the difference in install time was around 10x.

The only thing I had to do was switch from npm install to yarn. With a complex project like this you might want to use a temp package.json file that has all of the current package versions locked, e.g. =1.3.0 to create your new yarn.lock, then switch back to the original packge.json. I can try this whenever I have some free time.

A quick DDG search led me to this: https://docs.travis-ci.com/user/caching/#yarn-cache so you would be able to take advantage of caching with Yarn on Travis.

[ravichandranjv]

Not sure if this interjection is in context or not known already but isn't
it advised to put the package.json.lock file in Travis (or deployment)
rather than a temp file?

On Aug 4, 2017 11:19 PM, "Alessandro Dal Grande" [e-mail]
wrote:

[image: Boxbe] https://www.boxbe.com/overview This message is eligible
for Automatic Cleanup! ([email protected]) Add cleanup rule
https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3Dp%252B8nHPEIWcdsurRnysyXTtfE%252FmCMseoHAdWLKhYyfgo%253D%26token%3DTYxYBnECMjlcIu9N1kOdzTYhFUxDcmJftg1d1n8PsuQnWXJkw7kW9dvbrIuRrGCgkb1wlCqbHdAYKxKmuC9QtplrZ%252B7NZDmQICGuh3j1CvissxAwStK3j1zHyoc8cReDi0roF2JWMNC4r1kgcQg3Vw%253D%253D&tc_serial=31867441234&tc_rand=712407100&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001
| More info
http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=31867441234&tc_rand=712407100&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001

Unfortunately at this point I don't know much about this, but have seen
the difference on a few projects and the difference in install time was
around 10x. The only thing I had to do was switch from npm install to yarn.
With a complex project like this you might want to use a temp package.json
file that has all of the current package versions locked, e.g. =1.3.0 to
create your new yarn.lock, then switch back to the original packge.json.
I can try this whenever I have some free time.

A quick DDG search led me to this: https://docs.travis-ci.com/use
r/caching/#yarn-cache so you would be able to take advantage of caching
with Yarn on Travis.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mochajs/mocha/issues/2891#issuecomment-320310552, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAxH_OWbE2Gn-BflOmqJrIrw8BVvPVjxks5sU1g9gaJpZM4N9ama
.

[ScottFreeCode]

Definitely worth looking at, then; I'd be curious to see how Yarn and NPM 5 compare, given that the list of big improvements in NPM 5 was eerily similar to the list of advantages of Yarn over (previous versions of) NPM (or at least, it seemed that way going off of the general marketting buzz).

Something else that just crossed my mind -- we also could really use a solution specifically for PhantomJS and anything else that the global cache is only going to be storing, in effect, a downloader that still pulls the bulk of the package from the internet each time it's locally installed. I'm pretty sure PhantomJS alone takes up at least a tenth and probably more like a quarter of the Mocha dev install time.

[aledalgrande]

About PhantomJS, it seems that phantomjs-prebuilt can use the binary from disk, without downloading. Not sure about which kind of version is currently installed on Travis though.

If you don't change the package.json file often, another way would be to have a prebuilt Docker image and downloading that on Travis, completely skipping the packages install part.

[j-f1]

Travis notes that their cache is downloaded from an external storage solution, so it might not help too much if most of the time is spent downloading binaries.