Jest: Testing expected errors without Assertion Plan (misleading docs)

Created on 25 Oct 2016 · 7Comments · Source: facebook/jest

I was trying out Jest and started to write some tests around expected errors.
In the docs I found a section for async testing and error handling: https://facebook.github.io/jest/docs/tutorial-async.html#error-handling

// Testing for async errors can be done using `catch`.
it('tests error with promises', () => {
  return user.getUserName(3)
    .catch(e => expect(e).toEqual({
      error: 'User with 3 not found.',
    }));
});

// Or try-catch.
it('tests error with async/await', async () => {
  try {
    await user.getUserName(2);
  } catch (object) {
    expect(object.error).toEqual('User with 2 not found.');
  }
});

Both of these examples are super misleading, because the test passes in both cases if the function doesn't throw in the first place. E.g. this is a perfectly fine test that Jest doesn't complain about:

it('error handling', () => {
  try {
    console.log('definitely not failing')
  } catch (err) {
    expect(err).toBeTruthy()
  }
})

I couldn't find any hint in the docs whether something like plan(1) does or doesn't exist, but the migration guide for AVA and Tape mentions that it's unsupported. I would've loved to send a PR that improves the docs directly, but without a plan feature I don't see an elegant solution.

This isn't cool:

it('error handling', () => {
  const expect2 = jest.fn(expect)
  try {
    console.log('definitely not failing')
  } catch (err) {
    expect2(err).toBeTruthy()
  }

  expect(expect2.mock.calls.length).toBe(1)
  // fails now
})

This isn't cool either:

it('error handling', () => {
  try {
    console.log('definitely not failing')
    expect(false).toBeTruthy()
    // fails now
  } catch (err) {
    expect(err).toBeTruthy()
  }
})

And neither is this:

it('error handling', () => {
  try {
    console.log('definitely not failing')
  } catch (e) {
    var err = e
  }
  expect(err).toBeTruthy()
  // fails now
})

I'm not sure if it's just that I'm still thinking in terms of tap[e], or if this really is a problem, and how you would solve it elegantly in a Jest style.

Source

boennemann

All 7 comments

Yep, you are right we should improve this API and that it sucks currently. My hope was that we could make expect(() => {}).toThrow() async, but that's not easy either. I don't really like the "plan" functionality so much but we should potentially consider it – although it would be a pretty big hack into Jasmine. What do you think is best?

cc @dmitriiabramov

cpojer on 26 Oct 2016

Just a quick thought: If .toThrow() would always await the function passed to expect, would that work?

The same problem occurs for event emitters btw.

it('emit event', () => {
  emitter.on('event', (event) => {
    expect(event).to…
  })
})

In case the event is never fired the test still passes. I know that using a spy on the handler could make this work, but I think t.plan would be way easier in most cases.

boennemann on 26 Oct 2016

yes. this was always a very dangerous thing.
what if we introduce async matchers?

expect.async(await stuff).toThrowErrorMatchingSnapshot();

might also be a default behavior for expect to wait if a promise passed.

aaronabramov on 27 Oct 2016

I think plan does have some interesting use cases that I would love to see in Jest. There's a good article on it in the AVA docs.

In tape it's relatively easy to implement, because all expectations go through the t callback object like this example:

var test = require('tape');

test('timing test', function (t) {
    t.plan(2);

    t.equal(typeof Date.now, 'function');
    var start = Date.now();

    setTimeout(function () {
        t.equal(Date.now() - start, 100);
    }, 100);
});

Could you potentially do something similar in Jest, but using the global Jest object for the plan function like this?

it('does timing stuff', () => {
    Jest.plan(2);

    expect(typeof Date.now).toEqual('function');
    var start = Date.now();

    setTimeout(function () {
        expect(Date.now() - start).toBe(100);
    }, 100);
});

I don't know enough about how Jest is wired up behind the scenes yet to know if this would be practical. I'd love to have a go myself, but I haven't had much time to look at the source yet so might need a pointer on where to start looking.