Vstest: Option to minimize console output on vstest.console

Created on 22 Dec 2016 · 10Comments · Source: microsoft/vstest

Filing on behalf of @onovotny

Need a way to minimize output from running vstest.console (both desktop and coreclr). It currently lists the results of every test (Passed and failed) to the console and that’s very slow if you have a lot of tests.

There needs to be a way that only failed/skipped tests get logged to the console (perhaps with an error message). I’d prefer if this was the default but would settle for a command line switch (that was available to pass in via msbuild).

cli

Source

srivatsn

👍9

Most helpful comment

I think it's a great start towards the larger issue around having full control over the console input and output.

IMO, the default settings needs to be minimal by default, not normal. Most people don't use any extra parameters, they will run dotnet test. They will not know to add /logger:"console;verbosity=minimal" and even if they do, it's far to verbose to type out each time.

There's zero reason to output a Passing for a successful test by default, that's just noise (and slows down execution by outputting stuff to the console, which is slow).

clairernovotny on 24 Jan 2017

👍5

All 10 comments

If we don't show passed tests, it will be difficult to know whether a run is in progress or hung. May be we could show some progress marker and failed/skipped tests.

Would an option like /logger:"console;verbose=minimal" help? Other values of verbose could be all to show all test details (current default), quiet to show . for passed test and x for failed test.

codito on 23 Dec 2016

Proposal:

Add an option: /logger:"console;verbosity=minimal"
Two levels of verbosity are supported to start with:
- normal shows the existing experience
- minimal is detailed below
Default verbosity is normal

Minimal verbosity will look like following. It will only show Skipped and Failed test results.

> dotnet test --logger:"console;verbosity=minimal"
Starting test execution, please wait...
Failed   SampleUnitTestProject.UnitTest1.FailingTest
Error Message:
 Assert.AreEqual failed. Expected:<2>. Actual:<3>.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTest() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 29
Skipped  SampleUnitTestProject.UnitTest1.SkippingTest
Failed   SampleUnitTestProject.UnitTest1.FailingTestWithTraits
Error Message:
 Assert.Fail failed.
Stack Trace:
   at SampleUnitTestProject.UnitTest1.FailingTestWithTraits() in /home/codito/tmp/samples/mstesttest/Tests.cs:line 70

Total tests: 36. Passed: 33. Failed: 2. Skipped: 1.
Test Run Failed.
Test execution time: 3.2145 Minutes

/cc @pvlakshm @onovotny @piotrpMSFT

codito on 24 Jan 2017

I think it's a great start towards the larger issue around having full control over the console input and output.

There's zero reason to output a Passing for a successful test by default, that's just noise (and slows down execution by outputting stuff to the console, which is slow).

clairernovotny on 24 Jan 2017

👍5

This issue has been fixed via https://github.com/Microsoft/vstest/pull/416.
Currently the default behavior is normal, triage needs to take the final call. @codito @sbaid @pvlakshm.

singhsarab on 7 Feb 2017

+1 for defaulting to minimal verbosity with dotnet-test and hiding passing tests by default. The default output is overwhelming on any test project with anything more than 100 tests. It's unusable for test projects with more than a few hundred. We have test projects with thousands of tests.

natemcmaster on 7 Feb 2017

👍3

@singhsarab is this part of the 15.0.0-preview-20170210-02 release? I'm getting this error:

dotnet test --logger:"console;verbosity=minimal"

Error: Could not find a test logger with URI or FriendlyName 'console;verbosity=minimal'.

natemcmaster on 10 Feb 2017

@natemcmaster no. 20170210-02 is insertion from 15.0-rtm branch, this feature didn't meet the rtw bar.

Tagging @srivatsn @piotrpMSFT @livarcocc @sbaid if we want to do a bar check at this time.

codito on 11 Feb 2017

👍 for defaulting to v:m for console.

Besides, this is causing EntityFramework build on TravisCI to fail with error:

The log length has exceeded the limit of 4 MB (this usually means that the test suite is raising the same exception over and over).

The job has been terminated

See https://github.com/aspnet/EntityFramework/pull/7542#issuecomment-280090016

Please consider it for 15.0-rtm.