Karma: Improve karma to not always run the whole test suite

[NicBright]

Hi there,

my goal is to improve karma to not always run the whole test suite when a file changes.
To avoid lots of effort going in the wrong direction I would like to talk about this first, before starting the implementation.

Ideal Goal

Improve karma to not always run the whole test suite when a file changes

Motivation

This is an important issue in larger projects with a lot of tests. Changing a file (using autoWatch or grunt-karma) usually triggers a run of the whole test suite. This is not always necessary and can be avoided.

There are two isolated sets of files that usually trigger a re-run of all the tests:

1. source files
2. spec files

Regarding the first category one cannot do much easily. What one can do, is to derive the set of spec files four each source file from the first test run (maybe Istanbul can help here?). This is not what my proposal is about (although it should be done, but not as a first step).

What I want to improve is the second set of files: when changing a spec file, karma should run only this particular spec file

So let's talk about the ...

... Realistic Goal

Improve karma to not run the whole test suite when only a spec file changes

Proposed changes (i.e. backwards-compatible additions)

1. Add a sixth configuration option for the files pattern object (current keys are _pattern_, _watched_, _included_, _served_, _nocache_). It could be named isspec and defaulting to false. If set to true for file _A_, karma could (when the watcher detects a change in _A_) load (i.e. include as in the _included_ setting) only those files who have isspec == false plus _A_. Conversely, if the watcher detects a change in file _B_ which we assume to have isspec == false, then it would just load the files as usual.
2. Change No. 1 only works when running karma standalone with autoWatch option. It does not work in conjunction with grunt-karma (e.g.). Here, we can have a karma server started in background and tests are only invoked when the build-system detects a change (grunt-karma then uses the runner api to invoke a test run) **. The source code reveals: the whole config object is passed to the runner. But the runner does not respect all of its properties. The easiest solution would be to allow for a complete reconfiguration of _config.files_ (just using what is passed to the runner). However, this might affect performance, because if a developer switches between source and spec files (making changes), karma would have to re-do all the globbing again and again. I have the feeling that additional care has to be taken to ensure that this performance loss does not outweigh the benefits of switching between different _config.files_ configurations.

** The ideal setup I would like to support involves using (1) grunt-watch, setting up two different watch-targets, one for watching the source files and one for watching the spec files, and using (2) grunt-newer for the spec-files watch task to be able to dynamically pass only the changed files to the _grunt-files array_, e.g. like so: 'newer:karma:run'. Respecting the _grunt-files array_ was only "recently" (in January) made possible.

What do you think?

What do you think? If anything is unclear, I'm happy to try and explain more thoroughly.
I'm feeling that especially change No. 2 could be solved differently in many ways (e.g. letting grunt-karma do more of the work, like WebStorm does, for which the changes in runner.js were introduced in the first place, i.e. the changes to allow for notification on added/removed/changed files).

Regards,
Nicolas

[fbengrid]

Assuming this is meant in the context of someone working on the specific Spec file isn't the feature not already available via focused specs ( http://jasmine.github.io/2.3/focused_specs.html ) ?

Edit: I just realized that maybe you are not using jasmine... ignore my answer if that's the case.

[dignifiedquire]

Hey, first of all thanks for the extensive proposal. In regards to the second point you are suggesting, have you looked at #1468 ? As it seems to me this could potentially solve the issues you are mentioning here, or at least make things easier to handle.
In terms of actually running only specific spec files I think the idea of telling karma explicitly about this does make sense, but adding it as another option might make the api very complicated. Maybe we could split the files config into two parts like this:

files: {
  library: [
    'lib/**/*.js'
  ],
  tests: [
    'test/**/*.spec.js'
  ]
}

[NicBright]

@fbengrid: I'm well aware of this option, but as you've said, it is only available when using jasmine (and even there it's more of a hack: the angular team even wrote its own grunt task to ensure that no ddescribe or iit methods are accidentally committed ...)

@Dignifiedquire
Concerning extracting the watcher, I've just replied in that thread: https://github.com/karma-runner/karma/issues/1468#issuecomment-125212286

Regarding the files config. My first approach was similiar to your suggestion (adding a sibling config option like filesSpecs). But then I thought: isn't it easier to just setup the files array as usual? Using your suggestion, one would have to (1) write/support/maintain code that works both ways, while (2) maintaining the docs accordingly (either telling that both ways were supported, or introducing what looks like a breaking change in the docs (i.e. people coming from an old karma config file would think that their config was no longer valid).

To mitigate (1) one could deprecate the current usage sometime in the future. Which goes against backwards compatibility ...

Maybe it helps if we look at this change from the perspective of a small set of users (let's call them _crazy optimizers_) that try to optimize their dev environments as much as possible. Until now, developers have lived with their test suites running as a whole on every change ... so maybe they won't be missing much if this new setting was somehow "hidden" inside the files pattern object. The _crazy optimizers_ however will eventually find out about this option and be very happy. After all, not every project really needs this improvement, if they have only a small test suite ...

Conclusion

I agree, adding it as yet another option to the files pattern object makes it seem more complicated ("yet another possible option, very nested, very hidden"). This is justified, however by these arguments:

1. it's way easier to implement and thus easier to maintain
2. one stays backwards compatible forever
3. not everybody benefits from this option immediately (thus it's ok if some people miss it when setting up their dev environment for the first time)
4. thus, people who don't care will not see the increased difficulty that comes with adding the intermediary levels "library" and "tests", as they can just use files like it is today
5. Adding the intermediary levels "library" and "tests" does not cover every category of files, as (if I've understood the use-case for included: false, served: true correctly) sometimes one wants to serve other resources via karma, too (like images, css, ...).
6. even better implementations can be thought of: e.g. karma could somehow autodetect whether a file contains (almost entirely) specs or not (of course, this would be very expensive to implement as one would have to do this for the most important test frameworks ...). This would make it entirely unnecessary to specify which files were spec files.

[dignifiedquire]

@NicBright thanks for all the detailed work, I haven't forgotten you, just hadn't the time to go through it in detail.

[stuartaylward]

Protractor does this by specifying suites{} in the config file and then running a particular suite from the command line like 'Protractor --suite moduleX'.

I think it would be great for Karma to work the same way, it's easy to set up and run. But this is missing from Karma.

[NicBright]

@stuartaylward Your proposal does not really fit into this topic. This topic is about making karma itself smarter. Regarding your proposal: Karma in my understanding is "just" a test runner. It runs the tests you say it should run. So you can just use different karma config files (or different grunt tasks). One for each logical suit.

[saeidalidadi]

I have the same problem for my new project which is based on angularJs.
I have been solved this with my solution that I think would be useful to write it here.

My project directory structure is as follow:

│_ admin
│  │— dashboard
│  │   dashboard.module.js
│  │   dashboard.component.js
│  │   dashboard.specs.js
│  │— search 
│  │_ register
│
│— user
│— node_modules
│  karma.conf.js
│  webpack.config.js
│    ...
...
...

karma.conf.js

module.exports = function(config) {
   /**
    *  Env or Command Line arguments to find the `specs` for test
    *  Example: npm run admin:register  start just register component tests
    *  Example: npm run admin           starts all admin components tests 
    */
    config.set({
       files: [ /* test files */  ],
       preprocessors: {
          '*specs.js': [ 'webpack' ]
       }
    })
}

[jmjpro]

I'm also interested in the idea that @NicBright has presented here. Why hasn't there been any progress on this?

[IPWright83]

This would be a really interesting change. I've come here looking for an alternative to wallaby.js which is built entirely on this premise (quite successfully). If Karma were able to do this out the box (and even extend it in the future to run just impacted tests based on code coverage) it'd be a big step forward for TDD.

[xfg]

+1

[stefaneidelloth]

Related SO questions with workarounds using Gulp/Grunt to control Karma:

• https://stackoverflow.com/questions/26552729/karma-run-single-test/45350941#45350941

• https://stackoverflow.com/questions/28108485/getting-grunt-karma-to-run-one-unit-test

[mikeabiezzi]

+1

[CoderK]

+1

[stefaneidelloth]

I also first commented with "+1" but then I learned that this will spam all persons that already wrote a comment.
=> Please use the thumbs up feature on the issue description.

[AndrewEastwood]

+1

[emotec1]

+1

[macbem]

+1

[Drumstix42]

A property that lets us use a custom function to determine which file(s) to run tests on would be ideal, if it couldn't be easily automated.

For example:
**/*.spec.* file is the 'changed' file for the event. We detect that spec is part of the filename in our custom function and trigger tests to be performed for this file. Just need the karma runner to accept a specific file name.

And if we change another file, we could use the custom function to determine all related tests, or just run all of them if the spec file isn't what was updated. Etc etc etc.

Ultimately, I just need a way to trigger a test on a specific file or files once the karma server is running...