Jest: JUnit XML output for Continuous Integration w/ Jenkins

[palcu]

I have been trying to set up Jest on our Continuous Integration servers that run Jenkins. Even though Jest is built on top of Jasmine, I could not integrate it with jasmine-reporters.

I've used the branch that supports Jasmine 1.3. You can install it with npm install jasmine-reporters@~1.0.0.

My package.json has:

  "jest": {
    "setupTestFrameworkScriptFile": "<rootDir>/tests/unit/setup-jasmine-env.js",
...

And in setup-jasmine-env.js I have:

require('jasmine-reporters');

jasmine.VERBOSE = true;

jasmine.getEnv().addReporter(new jasmine.JUnitXmlReporter({
  savePath: "output/"
}));

The error I get is NodeJS attempt: Arguments to path.join must be strings or NodeJS attempt: Object [object Object] has no method 'join'. I've tried to debug it, but unfortunately you cannot set a debugger in a Jest test.

Is there a way I can make Jest work on Jenkins?

[jeffmo]

I suspect that you need to jest.dontMock() something...perhaps one of the node built-in modules?
What happens if you disable automocking right before require('jasmine-reporters')?

Also, in terms of debugging: The best way to do this is to run jest with --runInBand in order to force jest to run entirely inside a single process (rather than having jest boot a bunch of worker processes and having to guess which child process is running the test). Then you can use something like node-inspector to inspect the running process.

[palcu]

jest is undefined in setup-jasmine-env.js, because that is my setupTestFrameworkScriptFile which only has jasmine as a global variable.

I've made an example repo so you may debug faster the errors.

https://github.com/palcu/jest-example

As for debugging, I did not manage to set a single breakpoint in tests. I've installed node-inspector and used --runInBand, but it simply does not stop at the breakpoint.

[putermancer]

Another something interesting to add to the discussion. The line in jasmine-reporters that isn't working is an attempt to use path.join. Here's the relevant bit from the jasmine-reporters source code (console.log added for debugging):

var fs = require("fs");
var nodejs_path = require("path");
console.log('::WHAT IS PATH.JOIN::', nodejs_path.join);
console.log('::WHAT IS PATH::', nodejs_path);
var filepath = nodejs_path.join(path, filename);
var fd = fs.openSync(filepath, "w");
fs.writeSync(fd, text, 0);
fs.closeSync(fd);

As @palcu pointed out, the error is "Object [object Object] has no method 'join'". The output of the console statement is:

::WHAT IS PATH.JOIN:: undefined
::WHAT IS PATH:: {"Stats":"[Function: ]","exists":"[Function: ]","existsSync":"[Function: ]","readFile":"[Function: ]","readFileSync":"[Function: ]","close":"[Function: ]","closeSync":"[Function: ]","open":"[Function: ]","openSync":"[Function: ]","read":"[Function: ]","readSync":"[Function: ]","write":"[Function: ]","writeSync":"[Function: ]","rename":"[Function: ]","renameSync":"[Function: ]","truncate":"[Function: ]","truncateSync":"[Function: ]","ftruncate":"[Function: ]","ftruncateSync":"[Function: ]","rmdir":"[Function: ]","rmdirSync":"[Function: ]","fdatasync":"[Function: ]","fdatasyncSync":"[Function: ]","fsync":"[Function: ]","fsyncSync":"[Function: ]","mkdir":"[Function: ]","mkdirSync":"[Function: ]","readdir":"[Function: ]","readdirSync":"[Function: ]","fstat":"[Function: ]","lstat":"[Function: ]","stat":"[Function: ]","fstatSync":"[Function: ]","lstatSync":"[Function: ]","statSync":"[Function: ]","readlink":"[Function: ]","readlinkSync":"[Function: ]","symlink":"[Function: ]","symlinkSync":"[Function: ]","link":"[Function: ]","linkSync":"[Function: ]","unlink":"[Function: ]","unlinkSync":"[Function: ]","fchmod":"[Function: ]","fchmodSync":"[Function: ]","chmod":"[Function: ]","chmodSync":"[Function: ]","fchown":"[Function: ]","fchownSync":"[Function: ]","chown":"[Function: ]","chownSync":"[Function: ]","utimes":"[Function: ]","utimesSync":"[Function: ]","futimes":"[Function: ]","futimesSync":"[Function: ]","writeFile":"[Function: ]","writeFileSync":"[Function: ]","appendFile":"[Function: ]","appendFileSync":"[Function: ]","watch":"[Function: ]","watchFile":"[Function: ]","unwatchFile":"[Function: ]","realpathSync":"[Function: ]","realpath":"[Function: ]","createReadStream":"[Function: ]","ReadStream":"[Function: ]","createWriteStream":"[Function: ]","WriteStream":"[Function: ]","SyncWriteStream":"[Function: ]","lutimes":"[Function: ]","lutimesSync":"[Function: ]","lchown":"[Function: ]","lchownSync":"[Function: ]","lchmod":"[Function: ]","lchmodSync":"[Function: ]","FileReadStream":"[Function: ]","FileWriteStream":"[Function: ]"}

What is most interesting about this is that nodejs_path actually has the contents of the fs module, not the path module.

If you switch the require order around a little bit, the problem reverses itself and the fs variable ends up with the contents of the path module:

var nodejs_path = require("path");
var fs = require("fs");
console.log('::WHAT IS FS.OPENSYNC::', fs.openSync);
console.log('::WHAT IS FS::', fs);
var filepath = nodejs_path.join(path, filename);
var fd = fs.openSync(filepath, "w");
fs.writeSync(fd, text, 0);
fs.closeSync(fd);

Error: Object [object Object] has no method 'openSync'

Console output:

::WHAT IS FS.OPENSYNC:: undefined
::WHAT IS FS:: {"resolve":"[Function: ]","normalize":"[Function: ]","join":"[Function: ]","relative":"[Function: ]","sep":"/","delimiter":":","dirname":"[Function: ]","basename":"[Function: ]","extname":"[Function: ]","exists":"[Function: ]","existsSync":"[Function: ]"}

I've never seen an error quite like this one before, and the same code works without problems when run within node directly, jasmine-node, etc.

[palcu]

I've solved my first problem with debugging. I was using node-debug, but it does not support harmony. So I first have to open node-inspector & and then run the tests using node --harmony --debug-brk node_modules/jest-cli/bin/jest.js --runInBand. I think this should be documented somewhere.

I now also have JUnit XML output. Here is a sample repo. The only problems I have now is that I need to explicitly unmock path and fs in each test file because of https://github.com/facebook/jest/issues/106 and https://github.com/facebook/jest/issues/107.

[ghost]

Thank you for reporting this issue and appreciate your patience. We've notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.

[cpojer]

Please feel free to reopen this if there is anything in jest that needs to be changed to support this.

[megawac]

The issue I'm having with integrating jasmine-reporters is that because Jest spawns a different jasmine instance for each file, you cannot consolidate results. Normally you would want the test suites from all tests to be aggregated into a single output but with Jest it's only possible to create an output per file

[mitermayer]

Got this to work by adding to jest config

  ...
  "collectCoverage": true,
  "unmockedModulePathPatterns": [
    "./node_modules/jasmine-reporters"
  ],
 ...
"setupTestFrameworkScriptFile": `<rootDir>/jasmineReporter.js`,
 ...

the problem is that i also have on my config "collectCoverage": true, this creates the following output

 PASS  src/__tests__/index-test.js (0.549s)
2 tests passed (2 total in 1 test suite, run time 2.371s)
----------------------|----------|----------|----------|----------|----------------|
File                  |  % Stmts | % Branch |  % Funcs |  % Lines |Uncovered Lines |
----------------------|----------|----------|----------|----------|----------------|
 lib/test/            |      100 |      100 |      100 |      100 |                |
  jasmine-reporter.js |      100 |      100 |      100 |      100 |                |
 src/                 |    88.89 |      100 |    66.67 |    88.89 |                |
  index.ts            |    88.89 |      100 |    66.67 |    88.89 |              8 |
----------------------|----------|----------|----------|----------|----------------|
All files             |    92.31 |      100 |    66.67 |    92.31 |                |
----------------------|----------|----------|----------|----------|----------------|

Notice that the reporter is included in there, it should not be in there.

[mitermayer]

created a new issue with the above comment https://github.com/facebook/jest/issues/1274

[mitermayer]

got the same problem as @megawac , DId you manage to work around that ?

[megawac]

I did, I'll comment tmw (not with same library)

[megawac]

The only big gotcha of using jest with jasmine xml reporters is that because tests in different files are run in different contexts you can't use considateAll. So you have to merge the results after the tests are run.

jest/jasmine setup file for Junit results

var jasmineReporters = require('jasmine-reporters');
var reporter = new jasmineReporters.JUnitXmlReporter({
    consolidateAll: false,
    savePath: process.env.JUNIT_RESULTS,
    filePrefix: 'junit-'
});
jasmine.getEnv().addReporter(reporter);

Merge results scripts:

var fs = require('fs');
var path = require('path');
var async = require('async');

var reportMerger = require('junit-report-merger');

var results = fs.readdirSync(process.env.JUNIT_RESULTS);

let output = path.join(process.env.JUNIT_RESULTS, 'results.xml');
let files = results.map(x => path.join(process.env.JUNIT_RESULTS, x));
reportMerger.mergeFiles(output, files, {}, function() {
  async.each(files, fs.unlink);
});

[aaronabramov]

we've been working on our reporting system and now we can actually generate junit\xunit using a separate jest-reporter plugin. I think we're going to have a plugin for junit in a couple of weeks

[noahpeters]

do you have any examples of how to write or use a jest-reporter plugin?

[aaronabramov]

@noahpeters our internal reporters are here https://github.com/facebook/jest/tree/master/packages/jest-cli/src/reporters
but the thing is.. right now we don't have a mechanism to configure which reporters are used by jest without dealing with jest internals. As soon as we get that done, we'll be able to use any custom reporting system with jest

[kellyrmilligan]

any update on this for jest 15? I can't get junit xml output for the life of me! it's nice when a test fails in ci and what not.

[vampolo]

Any update on this beside the comment by @megawac https://github.com/facebook/jest/issues/104#issuecomment-233201678 ?

At the moment the missing integration with CI is a blocker to the adoption of jest in my org.

[cpojer]

There hasn't been any progress on this. Please feel free to send a PR to add this feature.

[rsolomon]

@vampolo What is blocking you from using the solution you linked above? The jasmine-reporters package accomplishes this with the more recent versions (I'm using 2.2.0).

Create a file, something like setup-jasmine-env.js, that performs the following:

const reporters = require('jasmine-reporters');
const reporter = new reporters.JUnitXmlReporter({
  // Jest runs many instances of Jasmine in parallel. Force distinct file output
  // per test to avoid collisions.
  consolidateAll: false,
  filePrefix: 'jest-junit-result-',
  savePath: __dirname + '/arc-jest-output.tmp/',
});
jasmine.getEnv().addReporter(reporter);

Then just reference it in your jest config:

"setupTestFrameworkScriptFile": "<rootDir>/jest/setup-jasmine-env.js",

Also make sure to ignore that file in your unmockedModulePathPatterns jest config.

[vampolo]

I was able to integrate that way at the end @rsolomon. No need for the unmockedModulePathPatterns tho since jest 15 does not automock anymore.

Still, this way of integrating junit output feel a bit hackish. Will try to hack a bit the jest codebase and see if i can come up with a PR.

Thanks all for the help.

[th3fallen]

@rsolomon you wouldnt happen to know of a way to only run the setup-jasmine-env (or generate junit tests) unless you're runing jest --coverage would you?

[nilliams]

I've just got this working thanks to that very useful help by @rsolomon , thanks!

@th3fallen The way I did that was to create a separate jest config file:

In ./jest/jest-junit.json:

{
    ...
    "setupTestFrameworkScriptFile": "./jest/setup-jasmine-env.js"
}

And then specify this config file when you want junit output (e.g. in your case along with --coverage):

jest --config ./jest/jest-junit.json --coverage

[danpan1]

Also you can try this node module for junit xml output https://www.npmjs.com/package/jest-junit

[kellyrmilligan]

I was able to get this working with the cli option as I didn't have access to change the config.

I then used junit-merge module to get just one file.

[marr]

@kellyrmilligan do you have an example? I'd like to use the new reporters configuration option, but I'd prefer to not write my own =)

[vampolo]

On the sideline here, but i ended up ditching junit format altogether and use the mocha formatting via jest-bamboo-formatter. So far it works pretty good the integration with Bamboo, but any mocha parser would work.

[kellyrmilligan]

create a script somewhere, chmod +x it:

const reporters = require('jasmine-reporters');
const path = require('path');
const reporter = new reporters.JUnitXmlReporter({
  // Jest runs many instances of Jasmine in parallel. Force distinct file output
  // per test to avoid collisions.
  consolidateAll: false,
  filePrefix: 'jest-junit-result-',
  savePath: path.join(process.cwd(), '/junit')
});
jasmine.getEnv().addReporter(reporter);

then however you're running it, something like this:

yarn run test:ci -- --setupTestFrameworkScriptFile=./bin/setup-jasmin-env.js

and after that is run, merge it all together:

node ./node_modules/.bin/junit-merge -d ./junit -o ./junit/merged-junit-results.xml

[lumio]

@kellyrmilligan ... that helped quite a lot! But you don't need junit-merge. Just report *.xml files