Jest: add option to fail tests that print errors and warnings

[harrykao]

Do you want to request a _feature_ or report a _bug_?

Feature.

What is the current behavior?

Tests pass even if they print errors and warnings to the console.

What is the expected behavior?

It would be nice if we could force the test to fail.

Our test suite prints lots of things like this:

    console.warn node_modules/moment/moment.js:293
      Deprecation warning: value provided is not in a recognized RFC2822 or ISO format. moment construction falls back to js Date(), which is not reliable across all browsers and versions. Non RFC2822/ISO date formats are discouraged and will be removed in an upcoming major release. Please refer to http://momentjs.com/guides/#/warnings/js-date/ for more info.

    console.error node_modules/fbjs/lib/warning.js:33
      Warning: Each child in an array or iterator should have a unique "key" prop.

(node:15601) UnhandledPromiseRejectionWarning: SyntaxError: The string did not match the expected pattern.

    console.error node_modules/fbjs/lib/warning.js:33
      Warning: `value` prop on `input` should not be null. Consider using an empty string to clear the component or `undefined` for uncontrolled components.

These are all legitimate errors that we should fix, yet our automated tests pass. If Jest could be made to fail, that would encourage developers to keep the tests clean.

[apaatsio]

This can already be achieved with a couple lines of code.

const spy = jest.spyOn(global.console, 'error')
// ...
expect(spy).not.toHaveBeenCalled();

[rickhanlonii]

@apaatsio is correct - the way to do it is something like that example

[chrisstevensap]

@apaatsio how would you suggest someone implement this so that it runs across all test suites without having to include it in each individual test suite?

[SimenB]

setupFiles, you can throw instead of assert

[lukeapage]

To get it working I've ended up with this ugly thing

beforeEach(() => {
    isConsoleWarningOrError = false;
    const originalError = global.console.error;
    jest.spyOn(global.console, 'error').mockImplementation((...args) => {
        isConsoleWarningOrError = true;
        originalError(...args);
    });
    const originalWarn = global.console.warn;
    jest.spyOn(global.console, 'warn').mockImplementation((...args) => {
        isConsoleWarningOrError = true;
        originalWarn(...args);
    });
});

afterEach(() => {

    if (isConsoleWarningOrError) {
        throw new Error('Console warnings and errors are not allowed');
    }
});

requirements:

1. still log the warning or error like it does currently
2. don't add a new stack trace to it
3. run automatically - don't have to add to every spec
4. still allow tests to mock console and add their own implementations and assert number of calls (this is why I can't just expect it to be called - our own logging code mocks console and tests the fn calls)

[lukeapage]

Note: I've tried lots of simpler ideas and they all fell down somehow.

[stevula]

Is there a preferred way to do this globally that doesn't involve adding a beforeEach/All or afterEach/All to every test file? I want all tests to fail if any console.error is called.

I'm looking at the solution from here, which seems to do the job:

let error = console.error

console.error = function (message) {
  error.apply(console, arguments) // keep default behaviour
  throw (message instanceof Error ? message : new Error(message))
}

[siddharthkp]

@stevula For what it's worth, that's exactly what works for me

[rickhanlonii]

Yup, that's the preferred way to do that globally

[akx]

Thanks, @stevula – that's close, but the problem is that the %s interpolation supported by the console isn't supported by thrown messages:

Error: Warning: Received `%s` for a non-boolean attribute `%s`.

If you want to write it to the DOM, pass a string instead: %s="%s" or %s={value.toString()}.%s

You can see the interpolated message too, but it's a little cumbersome. :)

[valgussev]

@akx as an option you can iterate through all the arguments and replace all possible substitutions strings with values, i.e.

console.error = (error, ...args) => {
  let errorMessage = typeof error === 'string' ? error : error.message;
  args.forEach(argument => {
    errorMessage = errorMessage.replace(/%(s|d|i|o|O)/, argument);
  });
  throw new Error(errorMessage);
};

[icaropires]

Is there a preferred way to do this globally that doesn't involve adding a beforeEach/All or afterEach/All to every test file? I want all tests to fail if any console.error is called.

I'm looking at the solution from here, which seems to do the job:

let error = console.error

console.error = function (message) {
  error.apply(console, arguments) // keep default behaviour
  throw (message instanceof Error ? message : new Error(message))
}

When I tried this option I got an error on executing because of Create React App:

So I wrote a bash script to for manually looking for consoles and warnings (it can be found here).

Later I discovered that options not supported by React Create App on jest configurations can be used via the command line. So one could run:

npm test -- --setupFiles './tests/jest.overrides.js'

instead of changing package.json, and the solution pointed by @stevula would work.

[scotthovestadt]

@icaropires If you want to disallow console.error (etc), I'd suggest doing it as a lint rule.

[icaropires]

@scotthovestadt Thanks for the suggestion. I did put it as a lint rule for my team's code. But in case of warnings, consoles were being called from inside my libraries:

This picture from my CI shows that.

[sdushay]

@lukeapage Thank you for contributing that solution (https://github.com/facebook/jest/issues/6121#issuecomment-412063935) -- just a note about it:

You're going to get Maximum Call Stack Size Exceeded when you call originalError/originalWarning, because you're still spying on the parent global.console object, so it recurses.

May I suggest either using console.log in their place if you want the whole stack trace, or just letting jest spit out the original error/warning to stdout, and throwing the error as usual?

See below for revised code:

let isConsoleWarningOrError;

beforeEach(() => {
  isConsoleWarningOrError = false;
  jest.spyOn(global.console, 'error').mockImplementation((...args) => {
    isConsoleWarningOrError = true;
    // Optional: I've found that jest spits out the errors anyways
    console.log(...args);
  });
  jest.spyOn(global.console, 'warn').mockImplementation((...args) => {
    isConsoleWarningOrError = true;
    // Optional: I've found that jest spits out the errors anyways
    console.log(...args);
  });
});

afterEach(() => {
  if (isConsoleWarningOrError) {
    throw new Error('Console warnings and errors are not allowed');
  }
});

[lukeapage]

@sdushay it does not recourse because once you grab a reference locally it is immune to spying. That’s why it’s calling original. The code above or a slight variation on it runs and works for us.

[sdushay]

@lukeapage You're right, I don't know what was blowing our call stack.

[greypants]

Here's the variant I landed on, using Node's Util.format function to preserve console string formatting via arguments in the error output:

// setupTests.js
import { format } from "util";

const error = global.console.error;

global.console.error = function(...args) {
  error(...args);
  throw new Error(format(...args));
};

cc @akx, re:

Error: Warning: Received %s for a non-boolean attribute %s.

[axelboc]

@greypants' version does not work for me with async tests. I'm trying to detect errors logged by React Testing Library - e.g. when a render is triggered after a test has completed. Throwing in console.error does not fail the async test that logs the error ... but fails the tests that follow it!

The version below, adapted from @sdushay, works nicely. It fails any async test that logs an error or warning to the console without affecting the others. Note that it is also compatible with ESLint's no-console rule and Jest's resetMocks option.

let consoleHasErrorOrWarning = false;
const { error, warn } = console;

global.console.error = (...args) => {
  consoleHasErrorOrWarning = true;
  error(...args);
};
global.console.warn = (...args) => {
  consoleHasErrorOrWarning = true;
  warn(...args);
};

afterEach(() => {
  if (consoleHasErrorOrWarning) {
    consoleHasErrorOrWarning = false;
    throw new Error('Console has error or warning');
  }
});