Nim: [unittest] test fail messages don't go to stderr

Created on 31 May 2018 · 13Comments · Source: nim-lang/Nim

Ref:

When a test fails, unittest sets the exit code to 1 (fail).

But whether the test passes or fails, the corresponding messages always go to stdout (errors don't go to stderr).

Stdlib

Source

kaushalmodi

Most helpful comment

Using both streams can cause confusion to some users who try to redirect the output in a naive way.

File streams are a very basic concept. It's important to understand the difference between stdout and stderr. The naive users should see the bash redirection basics of 1> and 2>.

kaushalmodi on 31 May 2018

👍2 👎1

All 13 comments

Because unittest uses styledEcho from the terminal module and styledEcho writes only to stdout.

data-man on 31 May 2018

Try with NIMTEST_NO_COLOR env var.

data-man on 31 May 2018

I tried that.

The envvar works as expected i.e. the messages don't have color any more. But the errors still come on stdout.

Here's how I confirm in Bash:

export NIMTEST_NO_COLOR=yes
# running compiled binary
./unitest_fail_test 1> stdout.txt
./unitest_fail_test 2> stderr.txt

The stderr.txt stays empty.

Here's the nim code for testing:

type
  Card = object
    rank: Rank
    suit: Suit
  Rank = enum
    crSeven
    crEight
    crNine
    crTen
    crJack
    crQueen
    crKing
    crAce
  Suit = enum
    csClubs = "♧"
    csDiamonds = "♢"
    csHearts = "♡"
    csSpades = "♤"
proc `<`(a,b: Card): bool = a.rank < b.rank
when isMainModule:
  import unittest
  suite "test card relations":

    setup:
      let
        aceDiamonds = Card(rank: crAce, suit: csDiamonds)
        kingClubs = Card(rank: crKing, suit: csClubs)
        aceClubs = Card(rank: crAce, suit: csClubs)

    test "greater than":
      check:
        aceDiamonds > kingClubs
        aceClubs > kingClubs
    test "equal to":
      check aceDiamonds == aceClubs

kaushalmodi on 31 May 2018

I also think that fails should be output to stderr, but let's wait other opinions.

data-man on 31 May 2018

👍2

I think the reported test failures is an essential output of the test program. It's not an error message (the test program didn't fail to complete its duties). Using both streams can cause confusion to some users who try to redirect the output in a naive way.

zah on 31 May 2018

If it's "not an error message", it shouldn't return an exit code of 1.

While setting exit code of 1, keeping stderr empty confuses the tool I am using.

Output to stdout if exit code is 0. Output to stderr if exit code is 1.

You need to output something to stderr in the case of an error, it could be even a simple message summarizing the errors.

kaushalmodi on 31 May 2018

👎1 👍1

Using both streams can cause confusion to some users who try to redirect the output in a naive way.

File streams are a very basic concept. It's important to understand the difference between stdout and stderr. The naive users should see the bash redirection basics of 1> and 2>.

kaushalmodi on 31 May 2018

👍2 👎1

No, it's just harmful friction, so now for every output statement I have to answer the question "should this go to stderr or stdout" and it doesn't solve anything because Unix's core assumption is that you can use line based parsing/filtering tools to connect programs together. Which never worked reliably and never will, but that's besides the point, stderr vs stdout is against Unix's core ideas.

Araq on 6 Jun 2018

No, it's just harmful friction, so now for every output statement I have to answer the question "should this go to stderr or stdout"

No. You can send all error messages to the stdout. But when ending the unittest, just send a summary error message to stderr saying that one or more of the tests in that unittest run failed. The issue is that while unittest exits with a code of 1, it creates no activity on stderr.

As I suggested earlier:

You need to output something to stderr in the case of an error, it could be even a simple message summarizing the errors.

kaushalmodi on 6 Jun 2018

@Araq

I have to answer the question "should this go to stderr or stdout"

Maybe, for unittests we could make exclusion.
Probably, it can be used for the tester: output only results with fails.

data-man on 6 Jun 2018

Or to add support for something like NIMTEST_FAILS_TO_STDERR env var.

data-man on 6 Jun 2018

No. You can send all error messages to the stdout. But when ending the unittest, just send a summary error message to stderr saying that one or more of the tests in that unittest run failed. The issue is that while unittest exits with a code of 1, it creates no activity on stderr.

I agree that's quite a nice solution for unittest, thanks for this idea.

(But in general stdout vs stderr is just another design bug in Unix, deal with it. ;-) )

Araq on 6 Jun 2018

unittest provides a structured output on stdout and the output of successful and failed tests should go in there to be readable and consistent.
Stderr is meant to be captured independently of stdout.
It should contain at least a 1-line summary in case of failure. IMO it should list the failed tests and, optionally, provide parsable output that can be easily displayed in CI.