Jest: Print list of failed tests at the end of test run

[hakunin]

Do you want to request a feature or report a bug?
Request a feature

What is the current behavior?
Jest spits out each error individually without any summary at the end of run about which tests failed (individual files)

What is the expected behavior?
Just like now + a helpful summary of which tests failed.

Example how RSpec does this (see the "Failed examples" part)

[cpojer]

Jest does this once you have enough test files for this to actually be useful, like 20 test suites or so.

[hakunin]

Got 35 suites, made 2 suits fail, no summary.

[jeduden]

@cpojer I got 5 suites. 12 tests total. Yet "pages" of logs. Being able to force a summary would be helpful.
Would you accept a PR that adds an option to force the summary ?

[ivyfae]

The output when run in cmd is longer than the buffer will keep for 6 failed tests. My experience is that this was something I really wished was implemented as soon as I wasn't able to see all the failed tests (so more than one) on the screen at the same time.

[hakunin]

@cpojer I believe this can be reopened since Jest doesn't print any summary?

[SimenB]

Threshold is set to 20: https://github.com/facebook/jest/blob/fb10f7a95161f3d93ec54be3b3f69359913b5691/packages/jest-cli/src/reporters/summary_reporter.js#L22

It seems to work fine for me, can somebody come up with a quick repro which shows the summary not being displayed?

An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first 🙂

[Fweeper]

@cpojer
@SimenB

The summary that it prints is not the summary that we're all looking for.

The current "summary" is _all_ of the output of all of the failed tests. Each failed test can have arbitrarily long output, which makes it very difficult to actually figure out the distinct _files_ that failed.

What we'd like in a summary is a single line per file (or perhaps per failing test), so that it's very easy to see exactly what failed without scrolling through hundreds or thousands of lines of text.

[SimenB]

That sounds like a custom reporter to me

[WickyNilliams]

I'd also like to voice my support for a summary of failing tests/suites in the output. I have 68 failing tests, over 22 suites. The current output is too long for my terminal buffer, so it's _very_ difficult to work through these failures (or even know which suites have failed!).

A short summary (simply a list of failing suites) at the end of the output would remedy this. I could then filter to a single suite at a time, fix, and move on. This workflow is unnecessarily difficult right now

[rickhanlonii]

Thanks for the bump @WickyNilliams, I think we can add a flag like compact which will show the suites that failed but not the failure message for each test

[WickyNilliams]

That would be amazing!

[Bessonov]

@SimenB An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first :slightly_smiling_face:

Is there any issue to improve that? Or is there any other way to show all suites and tests?

[Aprillion]

This is the only reasonable google result for jest do not repeat failed test details at the end, so I'm asking for help here:

I would like to see failure message 1 time for each test, not 2 times per failure - it's OK to list failed test in the summary at the end, but I would like to turn OFF the duplicated multi-page details of a snapshot mismatch (of a React component with a lot of data in props)...

Is there an existing CLI option please? Or another issue to add the feature?
(I am using the Jest version included in CRA, "react-scripts": "1.1.4", but we can update if the option is available in 1.1.5)

[sunfit]

Hey, so I ran in to the same issue and could not find a solution through google, so I ended up writing my own reporter jest-summarizing-reporter.
It can either print only the summary or print the test diffs at the top and then a summary.

Only when time came to publish to npm did I find out that at least 3 packages with same functionality already exist npm search results for jest summary reporters

If anyone finds this thread in same situation as I did, I hope this helps you.

P.S. There are at least 4 public implementations of this behaviour now and probably many more private ones.

Two top search results for jest summary reporter are this thread and one more where OP wants the same thing. Judging by these threads there is demand from the users.
Also if you look at the screenshot of the output of my reporter:

You can see that it's almost identical to the default one. The only real difference being that the failureMessage's are being printed before the full summary, not during it. So there is a lot of default reporter code duplication in the custom reporters - the colors, etc.

I honestly think this should be either a default behaviour or at least behind some option in jest.
I would love to try contributing if possible.

[SimenB]

Happy to take a PR making it an option passed to the default reporter (note that it should be reporters: [['default', {config: 'here'}]]) and not part of globalConfig

Docs for reporter config: https://jestjs.io/docs/en/configuration#reporters-array-modulename-modulename-options

[doniyor2109]

Hi @SimenB. Can I work on this issue?

[SimenB]

For sure, go for it! 🙂

[doniyor2109]

@SimenB How about adding this feature to summary reporter so that it will be shown at the end of the test results?

[sunfit]

@doniyor2109, @SimenB If I understand correctly that SummaryReporter is intended for the summary that is only shown when number of test suites is > 20 and there is at least one failing test, then that is not the place for this functionality.

The root problem is that when there are a lot of failed tests, their diffs fill the terminal and make it almost impossible to understand what were the tests that failed. This can happen without there being more than 20 suites.

What is actually needed is a list of names of failed tests, in a convenient and easy to access place (like the end of the output). That's it.

Removing the 20 suite threshold and the diffs (leaving only test names) from the SummaryReporter might be a good option. That way there would also be no diff duplication (which is present now when the SummaryReporter kicks in)

@doniyor2109 Please make sure you read all of the comments in this thread and in this one as well https://github.com/facebook/jest/issues/5358 and this denied PR https://github.com/facebook/jest/pull/5676 before starting. Or let me implement this, I mentioned in my comment above that I would not mind doing it, but you kinda beat me to it with your first comment.

[rickhanlonii]

As a user, I'd prefer keeping current summary at the end. It's a discrete overview of information that can be consistently expected at the end of the run, if we put anything after it that varies in length, then you will need to search to find the summary which is a bad experience IMO

It would probably be good to submit an overview of proposed changes before making any work to avoid churn

[llirikkk]

Threshold is set to 20:

https://github.com/facebook/jest/blob/fb10f7a95161f3d93ec54be3b3f69359913b5691/packages/jest-cli/src/reporters/summary_reporter.js#L22

It seems to work fine for me, can somebody come up with a quick repro which shows the summary not being displayed?

An option for forcing the summary might make sense, but I'd like to see if we can't make it smarter first 🙂

Having ability to force the summary definitely makes sense.

For example, I have a use case, when I need to include an excerpt of a console output from CI server and send it via email every time the build fails. That way I am able to get a quick feedback about what went wrong just by looking at the email.

But since the number of test suits in my project is lower than 20 and the size of the log excerpt included in the email is limited to a reasonably small amount (100 lines), I only get the log of passed tests, while failed tests are truncated.

With existing custom summary-reporter libraries I'm not able to do this, because they just print which tests failed, but doesn't print "expected vs actual" message.

I know it is a kind of specific case, but I think it might be useful in more general sense: when you need to get summary of failed test cases and the reason why they failed even if you have less than 20 test suites

[cefn]

If you are frustrated that a readable report listing failing suites is not possible out of the box with jest-cli, this bash workaround for Mac OS and probably linux, throws away all output except the suite failure lines...

npx jest 2>&1 | grep 'FAIL'

[aristofun]

Any ETA on this?

I wonder how people even use jest in real world without having summary in the end :(

[benwinding]

I found a little way to get the exact tests which failed:

Failed tests have this ● character at the beginning

Using @cefn's solution we can extract only the names of the _failed_ tests!

npx jest 2>&1 | grep '●'

---

Works pretty good

[Bessonov]

@benwinding no, this doesn't works well, because without splitting and combining output you get whole output xor overview, but not both.

[breck7]

The solution from @cefn worked great for me.

I added this to our package.json: "test-compact": "jest 2>&1 | grep 'FAIL'", and now can just run yarn test-compact on the command line to see a list of the failed tests. Failed tests are shown twice, but nbd.

[nbyavuz]

If you are frustrated that a readable report listing failing suites is not possible out of the box with jest-cli, this bash workaround for Mac OS and probably linux, throws away all output except the suite failure lines...

npx jest 2>&1 | grep 'FAIL'

Thanks for answer. Addition to that, you can use

npx jest 2>&1 | grep 'FAIL' | sort -t: -u -k1,1

for removing duplicate outputs.

[Nnadozie]

I think most people just use a Jest VScode extension, which shows failed tests inline and does more helpful stuff, but no one mentioned that here.