Karma: Karma 0.13.19 taking long time to complete when run via gulp

[mikelax]

I am opening a new ticket as this seems to be fallout from #1782 and the socket.io upgrade to 1.4.0.

I've just noticed a side effect from upgrading to 0.13.19; gulp karma test tasks take much longer to exit after the test run.

gulp.task('test:ng', function(done) {
  new Server({
    configFile: __dirname + '/karma.conf.js',
    singleRun: true
  }, done).start();
});

Previously the above task used to complete immediately after tests have completed. on 0.13.9, there is a pause of ~30 seconds after the karma runner completes the tests and the task exits.

gulp.task('test:ng', function(done) {
  new Server({
    configFile: __dirname + '/karma.conf.js',
    singleRun: true
  }, function() {
    console.log('done');
    done();
  }).start();
});

The above outputs done to the console but the gulp task still doesn't complete until after ~30 seconds.

cc @nazar

[dsine-de]

I'm currently implementing karma with gulp for the first time and can confirm this.
I was thinking karma wasn't exiting at all, but after reading this issue I waited ~30 secs and it exits correctly (but it takes very long).

[lukaselmer]

+1 caused by https://github.com/karma-runner/karma/commit/3ab78d6

[lukaselmer]

Until this issue is resolved, here's a simple workaround: Patch your package.json:

"devDependencies": {
    ...
    "karma": "=0.13.18",
    "socket.io": "1.3.7",
   ...
  },

[dignifiedquire]

This might also be caused by an internal change in socket.[email protected]

[austenLacy]

ditto here

[hamishtaplin]

Also having this problem.

[aragonezr]

Same problem here. [Mac (node 5.4.1, npm 3.3.12, gulp 3.9.0, karma 0.13.19)]
The patch mentioned by @lukaselmer works for me.

[mikezhuyuan]

Same issue. It works after changed to sockect.io 1.3.7 as mentioned by @lukaselmer. Please fix.

[BeneStem]

using socket.io 1.3.7 might not be a good advice cause it has a security issue that has been fixed with 1.4.0
but maybe you dont mind the security issue because you only use socket.io for dev

[therealklanni]

OK, I think I may have found the offending code. I can fork this and try the fix later, unless someone else gets around to it first.

Suspected offending commit: 3ab78d6

Suggested possible fix:

  for (var id in sockets) {
    if (sockets.hasOwnProperty(id)) {
      sockets[id].disconnect();
    }
  }

Also, this line seems suspect:

var sockets = socketServer.sockets.sockets // sockets.sockets? Is this correct?

Also, looking at the Socket.Io code, it seems there's already a disconnect method directly on the client instance that iterates through and closes all open sockets on the instance. Maybe a good idea to look into just calling that rather than reimplementing the same functionality in Karma?

Hazarding a guess (can potentially replace Karma's workaround?):

socketServer.disconnect(); // and.... done!

Thoughts welcomed.

Side note: To me this seems like a silly way to use Object.keys anyway, and maybe for..in should be preferred all around? Just a thought.

[dignifiedquire]

@therealklanni I wish it was that simple, but sadly it isn't if you read the the git history and the comments surrounding that code you see that there are issues inside socket.io that we are working around, and some in our own code which makes it right now necessary to hack into the sockets list and disconnect them manually.

I believe if https://github.com/socketio/socket.io/issues/2069 gets fixed we should be able to fix this properly but I don't see a good way to workaround this at the moment.

[lukaselmer]

Seems like the issue is resolved by using socket.io 1.4.5 https://github.com/socketio/socket.io/commit/b3fc530abefd384b3a89ff5493e97f3ef85098d4! Thanks!

[anthonyhastings]

@lukaselmer - Good spot! I've just tried this out locally and can confirm that by bumping the version of socket.io to 1.4.5, this issue disappears for me.

With that in mind, is the only actionable thing here required for Karma, to release a new minor version with the bumped version?

[dignifiedquire]

You can just run npm update on your installation, as 1.4.5 is in the dependency range of karma already. There is version bump in master but that will not be released immediately due to some other things I'm waiting for

[nazar]

We are still seeing this issue, specifically when running Gulp with Karma and phantomjs. There is a 30 second delay between tests completing and the Gulp process exiting when upgrading from karam: 0.13.18 to 0.13.19:

Using karma: 0.13.19 with socket.io: 1.4.5

vagrant@vagrant-ubuntu-trusty-64:~/files$ time gulp test:ng
[12:23:54] Using gulpfile ~/files/gulpfile.js
[12:23:54] Starting 'test:ng'...
01 02 2016 12:23:54.809:INFO [karma]: Karma v0.13.19 server started at http://localhost:9876/
01 02 2016 12:23:54.813:INFO [launcher]: Starting browser PhantomJS
01 02 2016 12:23:54.968:INFO [PhantomJS 2.1.1 (Linux 0.0.0)]: Connected on socket /#VODKn20_IF-mzhLIAAAA with id 61199889
 PhantomJS 2.1.1 (Linux 0.0.0): Executed 62 of 62 SUCCESS (0.575 secs / 0.238 secs)
.
PhantomJS 2.1.1 (Linux 0.0.0): Executed 62 of 62 SUCCESS (0.575 secs / 0.238 secs)
[12:23:56] done
[12:23:56] Finished 'test:ng' after 1.75 s

real    0m33.284s
user    0m3.229s
sys 0m0.265s

Using karma: 0.13.18 with a package.json explicit socket.io: 1.3.7

vagrant@vagrant-ubuntu-trusty-64:~/files$ time gulp test:ng
[12:37:13] Using gulpfile ~/files/gulpfile.js
[12:37:13] Starting 'test:ng'...
01 02 2016 12:37:13.956:INFO [karma]: Karma v0.13.18 server started at http://localhost:9876/
01 02 2016 12:37:13.960:INFO [launcher]: Starting browser PhantomJS
01 02 2016 12:37:14.113:INFO [PhantomJS 2.1.1 (Linux 0.0.0)]: Connected on socket HRp9tkc4qokBmHjgAAAA with id 81794516
 PhantomJS 2.1.1 (Linux 0.0.0): Executed 62 of 62 SUCCESS (0.57 secs / 0.233 secs)
.
PhantomJS 2.1.1 (Linux 0.0.0): Executed 62 of 62 SUCCESS (0.57 secs / 0.233 secs)
[12:37:15] done
[12:37:15] Finished 'test:ng' after 1.75 s

real    0m3.585s
user    0m3.242s
sys 0m0.249s

The Gulp task is:

gulp.task('test:ng', function(done) {
  new Server({
    configFile: __dirname + '/karma.conf.js',
    singleRun: true
  }, function() {
    gutil.log('done');
    done();
  }).start();
});

done is logged to the console right after the test run completes but Gulp exists 30 seconds later. This can be overcome by adding a process.exit() after the log statement but that breaks our gulpfile task chains - i.e. cleanup tasks after the test is run.

The above Gulp task completes and exits immediately when using karma: 0.13.18 with socket.io: 1.3.7

For testing this issue I can confirm socket.io: 1.4.5 is installed along with karma: 0.13.19

[email protected] node_modules/karma
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected])
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
├── [email protected]
└── [email protected]

vagrant@vagrant-ubuntu-trusty-64:~/files$ uname -a
Linux vagrant-ubuntu-trusty-64 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

vagrant@vagrant-ubuntu-trusty-64:~/files$ node --version
v4.2.4

vagrant@vagrant-ubuntu-trusty-64:~/files$ gulp --version
[12:32:27] CLI version 3.9.0
[12:32:27] Local version 3.9.0

vagrant@vagrant-ubuntu-trusty-64:~/files$ node_modules/.bin/phantomjs --version
2.1.1

Please let me know if I can add any further version numbers or information.

[anthonyhastings]

I've seen this myself, that after pointing at socket.io: 1.4.5 every say, one in six runs of my gulp task it'll still hang for those 30 seconds.

[deepsweet]

same here, we are still experiencing 30s delay problem even with updated
socket.[email protected].

On Monday, 1 February 2016, Anthony Hastings [email protected]
wrote:

I've seen this myself, that after pointing at socket.io: 1.4.5 every say,
one in six runs of my gulp task it'll still hang for those 30 seconds.

—
Reply to this email directly or view it on GitHub
https://github.com/karma-runner/karma/issues/1788#issuecomment-177956732
.

[dignifiedquire]

I've just tested this and I'm afraid I'm not able to reproduce. What I'm using is the following:

$ git clone https://github.com/karma-runner/gulp-karma.git
$ cd gulp-karma
$ npm i
$ gulp test

This is working fine as expected on a fresh install. Could you please check if that is the case for you as well?

[nazar]

Super thanks for the update @dignifiedquire!

I cloned git clone https://github.com/karma-runner/gulp-karma.git as per your comments and gulp test:

vagrant@vagrant-ubuntu-trusty-64:~/test/gulp-karma$ gulp test
[14:05:22] Using gulpfile ~/test/gulp-karma/gulpfile.js
[14:05:22] Starting 'test'...
01 02 2016 14:05:22.762:INFO [karma]: Karma v0.13.19 server started at http://localhost:9876/
01 02 2016 14:05:22.766:INFO [launcher]: Starting browser Chrome
01 02 2016 14:05:22.767:ERROR [launcher]: No binary for Chrome browser on your platform.
  Please, set "CHROME_BIN" env variable.
[14:05:22] 'test' errored after 47 ms
[14:05:22] Error: 1

I updated package.json by adding "karma-phantomjs-launcher": "1.0.0":

vagrant@vagrant-ubuntu-trusty-64:~/test/gulp-karma$ cat package.json
{
  "name": "gulp-karma",
  "version": "0.0.1",
  "description": "",
  "repository": {
    "type": "git",
    "url": "https://github.com/karma-runner/gulp-karma"
  },
  "bugs": {
    "url": "https://github.com/karma-runner/gulp-karma/issues"
  },
  "scripts": {
    "test": "gulp test"
  },
  "homepage": "https://github.com/karma-runner/gulp-karma",
  "devDependencies": {
    "jasmine": "^2.4.1",
    "karma": "^0.13.0",
    "karma-jasmine": "^0.3.6",
    "karma-chrome-launcher": "^0.2.0",
    "karma-phantomjs-launcher": "1.0.0",
    "gulp": "^3.9.0"
  }
}

and set karma.conf.js as follows:

vagrant@vagrant-ubuntu-trusty-64:~/test2/gulp-karma$ cat karma.conf.js
module.exports = function(config) {
  config.set({
    browsers: ['PhantomJS'],
    frameworks: ['jasmine'],
    files: [
      'src/**/*.js',
      'test/**/*.spec.js'
    ]
  });
};

The first few runs completed very quickly but as @antwan1986 mentioned, my fifth run experienced 30 seconds delay in exiting:

vagrant@vagrant-ubuntu-trusty-64:~/test2/gulp-karma$ time gulp test
[14:24:23] Using gulpfile ~/test2/gulp-karma/gulpfile.js
[14:24:23] Starting 'test'...
01 02 2016 14:24:23.107:INFO [karma]: Karma v0.13.19 server started at http://localhost:9876/
01 02 2016 14:24:23.112:INFO [launcher]: Starting browser PhantomJS
01 02 2016 14:24:23.268:INFO [PhantomJS 2.1.1 (Linux 0.0.0)]: Connected on socket /#cskgD06k6P0nWcUuAAAA with id 18468465
PhantomJS 2.1.1 (Linux 0.0.0): Executed 1 of 1 SUCCESS (0.039 secs / 0.001 secs)
[14:24:23] Finished 'test' after 289 ms

real    0m30.878s
user    0m0.725s
sys 0m0.049s

There was a pause of ~ three seconds between each test run.

Although I'm seeing the 30s delay intermittency in this example, I am experiencing the 30s delay 100% of the time when running our tests.

Again, HTH and please let me know if there is any other information or tests I can run to help track this down.

[dignifiedquire]

Interesting, this sounds to me like it is related to phantomjs in this instance, could you please check with another browser (Firefox or Chrome) if the issue happens there as well for you?

[nazar]

Hey @dignifiedquire,

Thank you for your continued assistance in tracking this down!

could you please check with another browser (Firefox or Chrome) if the issue happens there as well for you?

I was not able to install either of chrome not chromium inside our headless Vagrant Ubuntu based VM, which is where the tests currently run - this mirrors our CI server.

I can confirm that I don't experience the 30 second delays running your test repo with Chrome under OSX 10.11.3.

Interestingly enough, when I revert phantomJS on our VM back to 1.9.8 using "phantomjs": "1.9.19" in package.json I can no longer reproduce the intermittent 30 second delay.

It is starting to look like PhantomJS 2.x is responsible for this issue, at least in our test setup which is installed as a karma-runner/karma-phantomjs-launcher 1.0.0 version dependency on "phantomjs-prebuilt": ">=1.9" which is pulling in PhantomJS 2.1.1

For us, the current solution is to explicitly require PhantomJS 1.9.19 and to revert karma-runner/karma-phantomjs-launcher back to 0.2.3 as follows

"phantomjs": "1.9.19",
"karma": "0.13.19",
"karma-phantomjs-launcher": "0.2.3",

It would be interesting to hear if anybody else experiencing the 30 second delay issues is also using PhantomJS 2.x

[przemcio]

@dignifiedquire The issue is definitely related to phantomJS 1.9.x. There isn't any time delay problem while I used Firefox and/or Chrome.

[henryptung]

@nazar Seeing the same thing here.

After doing some digging:

The 30 second timeout is from https://github.com/websockets/ws/blob/master/lib/WebSocket.js#L129, where if the WebSocket is closed uncleanly (without close acknowledgement from the client), the socket goes into limbo mode, waiting to receive this ack before fully closing out.

It seems like Karma, in singleRun mode, will try to kill the browser immediately upon browser_complete, rather than first closing the socket and waiting for completion of socket close. Ultimately, Karma will try to push some buffered information down the socket (trying to inform the browser of its own startup, actually), which hits EPIPE, and triggers this delayed close mode of the socket, keeping the Karma process alive an additional 30 seconds.

Regarding PhantomJS versions, I was able to reproduce this both with PhantomJS 2.x and with PhantomJS 1.9.8 + karma-phantomjs-launcher 0.2.3. Current workaround is to revert to karma 0.13.9, karma-phantomjs-launcher 0.2.1 which we were previously using.

[keirlawson]

This bug is still there for me using Karma 0.13.21, downgrading to 0.13.9 resolves

[r-park]

I've experienced the bug in 0.13.21 as well. Downgraded to 0.13.18

[resistdesign]

@keirlawson Locking to [email protected] worked! :)

As a side note, [email protected] gave me socket related errors.

[boschni]

All versions from 0.13.10 and 0.13.22 seem to contain the bug. We also downgraded to 0.13.9.

[jasonolmstead33]

Still seeing this issue. Downgrading to 0.13.9 is an issue because it doesnt like node 4.3...

im seeing:
[email protected]: wanted: {"node":">=0.10 <=0.12 || >=1 <=3"} (current: {"node":"4.3.0","npm":"2.14.12"})

[donaldpipowitch]

I can confirm this. I'm not using gulp, but Karmas Node API. I switched back to [email protected] (and I use Node v4.2.1) and everything works fine now.

[johannesjo]

Still an issue for me even when using karma 0.13.9 and karma 0.13.22.

[arshaw]

@danielpacak is attempting to update the gulp karma docs. his technique works well:
https://github.com/danielpacak/gulp-karma/blob/30aa28b3b6a0c9321b16a4ee174fe140b51d9ff3/README.md
(see the PR)

[fegemo]

I am also seeing this on version 1.1.1.

[johannesjo]

The technique suggested by @arshaw works if you only want to start a single run, but tasks you defined to run afterwards will not be executed.

EDIT: At least that is the case for me, when using it together with run-sequence. The reason for this being, that if there is no done callback function provided to new KarmaServer() as second parameter, process.exit is being called instead by karma. What I tried to avoid this is this:

gulp.task('testSingle', function(done) {
    new KarmaServer({
        configFile: __dirname + '/../karma.conf.js',
        singleRun: true
    }, function() {
        // provide an empty callback function so process.exit
        // is not called by karma
    })
        .on('run_complete', function(browsers, results) {
            done(results.error ? 'There are test failures' : null);
        })
        .start();
});

This however put me back to square one. Now the process isn't exiting again, after completion.

[Dinistro]

I also had this issue. I was able to fix it with a workaround provided here: https://github.com/karma-runner/gulp-karma/pull/23#issuecomment-232313832

[gandra]

Having the same issue with fresh new angular2 app made with angular cli: http://stackoverflow.com/questions/42030568/how-to-manage-to-exit-phantomjs-launcher-after-tests-execution

[Koslun]

Still seeing this issue with karma version 1.4.1. I.e. after the gulp tasks have completed and results have been output to console there is a roughly ~35 second delay before the process fully exits. Manually counted to around ~35 seconds on my local relatively powerful Ubuntu desktop machine while the pipelines I've seen on our CI on Codeship repeatedly got around the same value of 35 seconds +/- 2 seconds maybe.

Quick-fix using https://github.com/karma-runner/gulp-karma/pull/23#issuecomment-232313832 as linked by @Dinistro. karma was however not correctly resolved and so had to explicitly resolve the it from node_modules. Full example below:

const gulp = require('gulp');  // @4.0.0-alpha.2
const gutil = require('gulp-util');  // @3.0.7:

gulp.task('karma:single-run', karmaSingleRun);

function karmaSingleRun(done) {
  const childProcess = require('child_process');
  childProcess.exec('./node_modules/karma/bin/karma start conf/karma.conf.js', (err, stdout) => {
    gutil.log(stdout);
    if (err) {
      throw new Error('There are test failures');
    } else {
      done();
    }
  });
}

Curious if this still or again has something to do with the manual socket disconnects that Karma does, which was discussed earlier in this thread. Or if it is something upstream like the tag suggests.

Do we need to create a new issue or will this be re-opened?

EDIT: Previous version of course does not run on Windows. Here is a version that works on Windows as well (though it also requires an additional gulp library):

const gulp = require('gulp'); // @4.0.0-alpha.2
const shell = require('gulp-shell'); // @0.5.2
const gutil = require('gulp-util'); // @3.0.7:

gulp.task('karma:single-run', shell.task([
  'karma start <%= configFile %>'
], {
  verbose: true,
  templateData: {
    configFile: configFile
  }
}));

[yoweiner]

@nazar I'm having this issue with PhantomJS 2.1.1, but reverting to 1.9.7 (by substituting it on the PATH and PHANTOMJS_BIN at runtime) fixes it.
[email protected]
[email protected]