Vstest: Improve readability of dotnet test

[livarcocc]

_From @terrajobst on October 22, 2018 1:21_

The command line output of dotnet test is extremely noisy which makes it hard to parse. Instead of this:

$ dotnet test
Build started, please wait...
Build completed.

Test run for P:\minsk\Minsk.Tests\bin\Debug\netcoreapp2.1\Minsk.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.6522298]     Minsk.Tests.CodeAnalysis.Syntax.ParserTests.SyntaxFacts_Kind_AgreesWith_GetText [FAIL]
Failed   Minsk.Tests.CodeAnalysis.Syntax.ParserTests.SyntaxFacts_Kind_AgreesWith_GetText
Error Message:
 Assert.Equal() Failure
          ↓ (pos 0)
Expected: immo
Actual:   brad
          ↑ (pos 0)
Stack Trace:
   at Minsk.Tests.CodeAnalysis.Syntax.ParserTests.SyntaxFacts_Kind_AgreesWith_GetText() in P:\minsk\Minsk.Tests\CodeAnalysis\Syntax\ParserTests.cs:line 32

Total tests: 65. Passed: 64. Failed: 1. Skipped: 0.
Test Run Failed.
Test execution time: 1.2981 Seconds

I wish the output would be like:

Minsk.Tests\Minsk.Tests.csproj (.NETCoreApp,Version=v2.1)

ParserTests.SyntaxFacts_Kind_AgreesWith_GetText
    Assert.Equal() Failure
                  ↓ (pos 0)
        Expected: immo
        Actual:   brad
                  ↑ (pos 0)
    at Minsk.Tests.CodeAnalysis.Syntax.ParserTests.SyntaxFacts_Kind_AgreesWith_GetText() in P:\minsk\Minsk.Tests\CodeAnalysis\Syntax\ParserTests.cs:line 32

Total tests:  65
     Passed:  64
     Failed:   1
    Skipped:   0
       Time: 1.3 secs

Requests like #10165 are related and are an indicator that folks working on the command line want legible output.

Be readable. Be brief. Be gone.
-- My opinion on good output.

_Copied from original issue: dotnet/cli#10207_

[livarcocc]

@KathleenDollard @kendrahavens

[livarcocc]

Should probably move this issue to microsoft/vstest.

[khellang]

A little (more) color (when supported) would probably help a lot as well 👍🏻

[Cronan]

Would be great if this command had an option to output in a more structured format, e.g. XML and/or JSON as an option - not many real standards in this space, TAP?

[bradwilson]

I see what's going on here...

        Expected: immo
        Actual:   brad

😝

[adamralph]

I'm all for making the test output less noisy, but

Total tests:  65
     Passed:  64
     Failed:   1
    Skipped:   0
       Time: 1.3 secs

is _more_ noisy than

Test Run Failed. Total tests: 65. Passed: 64. Failed: 1. Skipped: 0. Time: 1.3 secs

If I run 10 test suites, I'd much rather see 10 lines than 50 lines.

Also, a problem with multi-line output, where the meaning of one line is dependent on the meaning of the previous, is that it completely breaks down when running test suits in parallel, e.g.

Total tests:  132
Total tests:  65
     Passed:  64
     Failed:   1
     Failed:   0
    Skipped:   0
     Passed:  132
    Skipped:   0
       Time: 2.1 secs
       Time: 1.3 secs

[adamralph]

And specifically, IMO the new output in 2.2.300 is at least partially a regression, i.e. in older versions:

Total tests: 65. Passed: 64. Failed: 1. Skipped: 0.

in 2.2.300:

Total tests:  65
     Passed:  64
     Failed:   1
    Skipped:   0

This is now worse than before.

[adamralph]

Moreover, it would be great if the result line could repeat the test project and platform:

Test Run Failed. Total tests: 65. Passed: 64. Failed: 1. Skipped: 0. Time: 1.3 secs. P:\minsk\Minsk.Tests\bin\Debug\netcoreapp2.1\Minsk.Tests.dll(.NETCoreApp,Version=v2.1)

[KathleenDollard]

It could drop a few more words, and is the full directory interesting?

Consider playing with the verbosity settings so folks can tune this if @adamralph 's opinion isn't widespread.

Failed! Total: 65. Passed: 64. Failed: 1. Skipped: 0. Time: 1.3 secs. P:\minsk\Minsk.Tests\bin\Debug\netcoreapp2.1\Minsk.Tests.dll(.NETCoreApp,Version=v2.1)

[adamralph]

@KathleenDollard true, we could probably drop the path, since two test assemblies with the same name in different paths is an edge case. We could also use the TFM for the platform:

Failed! Total: 65. Passed: 64. Failed: 1. Skipped: 0. Time: 1.3 secs. Minsk.Tests.dll (netcoreapp2.1).

[adamralph]

Consider playing with the verbosity settings...

How about we have the single line for minimal verbosity and multiple lines for normal?

[KathleenDollard]

Coming back to this, I find this hard to parse. Maybe (and I'm not on that team, just offering an opinion)

Failed! Fail/Pass/Skipped/Total:    1   64    0   65. (1.3 secs) Minsk.Tests.dll (netcoreapp2.1).

[adamralph]

@KathleenDollard personally I find that list of whitespaced numbers too difficult to tie back to Fail/Pass/Skipped/Total.

My current vote is for:

--verbosity minimal

Failed! Total: 65. Passed: 64. Failed: 1. Skipped: 0. (1.3 s) Minsk.Tests.dll (netcoreapp2.1).

or if we loosen the grammar slightly:

Failed! Total: 65. Pass: 64. Fail: 1. Skip: 0. (1.3 s) Minsk.Tests.dll (netcoreapp2.1).

(The critical aspect being minimal = one line.)

And at higher verbosity I don't really care:

--verbosity normal

go
nuts
with
as
many
lines
as
you
want

[drognanar]

Tagging @cvpoienaru

[Sanan07]

It looks like all work is done her. Closing.