Vstest: Improve the cli experience for dotnet test

[singhsarab]

Description

There has been a lot of feedback around the cli experience for dotnet test, here are a few suggestions that can be considered.

o Show the results using unicode ✔ and ❌ with colors
o Progress bar (only prints a dot every ~3 seconds just so the user knows we haven’t frozen.)
o Clean summary table at the end of run for the whole solution file. Includes Pass/Fail Unicode character, Test Name, and duration.
o Option to output an html file.

[akdor1154]

• when running dotnet test in a non-test project, try to find the nearest solution and run with respect to that

[jimmylewis]

Copying some feedback from a VS Feedback ticket:

After running tests in the console, it used to be really easy to search for “Passed”, “Skipped”, and “Failed” tests using those strings. When I used to get a result like:

Test Run Failed.
Total tests: 6543
Passed: 6502
Failed: 1
Skipped: 40
Total time: 4.7696 Minutes

I could find the failing test just by searching in the console for "Failed".

Now the output uses ✔, !, and X instead of these. X is much more commonly found in test names or output messages (e.g. in “exception” messages or test names containing words like syntax, index. matrix - all of which are in my test bed), so searching becomes less effective to find the test failure.

Worth noting, I couldn't search for ❌ in the VS dev cmd prompt. I had to use the letter X, which did find the match in the console contents.

[jpribele]

Unicode Pass/Fail is bad for farms or other processes are that are capturing tty.
û TestName example output

[nohwnd]

(in reaction to PR #2333)

Personally I find the default indicators really nice to look at, but a pain to read and search for in log. I like the change but I don't think adding yet another set of options is the best way to do this.

Maybe we could consider changing them:

• to something that is a normal character / character group, but is unlikely to turn up in log otherwise. (e.g. use [-], [+], [!], but not just +,-,! or X).
• put the indicators to the header, like Running tests (Passed √, Failed X, Skipped !).
• use Passed, Failed, Skipped instead of cryptic signs

~Another problem is that the characters easily get mangled in output, but maybe that is just our problem, because we are likely the only ones running vstest in vstest (in our acceptance tests).~ I see the same issue mentioned above, so there not using unicode would be an advantage as well.

[jpribele]

I could live with both unicode and plain text. [Passed √] TestName
In log file that has 2000 tests being able search quickly and reliably is nice. Ctrl+F "[Failed" or "[Passed

[nohwnd]

For what it's worth searching for     X  (four spaces X and one space), will find failed tests pretty consistently, especially if you switch to the raw log view.

(Also passed tests are showing � PassingTest2 [7ms] in the raw view)

[singhsarab]

@nohwnd This was based on the feedback shared by @kendrahavens. Please get this reviewed from her. Thanks.

[kendrahavens]

I like @nohwnd 's suggestion of using [-], [+], [!]. Overall, the goal was to make it easier to distinguish between the test states at a glance. Icons with color tend to provide faster recognition than the spelled out 'Passed' and 'Failed' states. But since the unicode creates issues for searching I'd agree we can change to a more easily searchable character group.

@jpribele Would the character group [-], [+], [!] also be satisfactory?

[nohwnd]

I made a quick poll on twitter to involve more people into this.

[jpribele]

Plus Minus is fine.

[duaneking]

Just want to point out in as respectfully a tone as I can use while at the same time wanting to drive home the importance of this, that using emoji of ANY KIND violates accessibility standards and is not accessible for the many visually disabled or older software developers who are still very much a part of the software development community and reply on braille and other terminal emulation that do not support emojis BY DESIGN.

Please don't use emoji for ANYTHING by default as it can crash text to speech systems used by the blind or disabled and is generally not accessible anyway so making this the default can activly harm people or limit what they can contribute to.

Allow it via an optional flag maybe, but you should be as inclusive as possible and keep to things like the ADA, etc so it remains legal.

[nohwnd]

@duaneking Thank you for your concern 🙂 We will not change the current approach to emojis, in fact it will change back to the Passed, Failed etc. text.

I am not sure if you took part of the poll above but the majority of voters voted for the text to return, and I think it is the best option as well:

[duaneking]

I'm not able to see or reply to this due to errors in the oauth provider used by github, it wont let me log in.  Please reopen.

Thanks, - Duane King

[nohwnd]

@duaneking Ithink all the above points were addressed. I just did the last remaining part which is reverting the indicators back to words. That is why this is closed.

[duaneking]

        Thank you, right now my only ability to answer any of these is through email by responding there because I’m not able to login and view the actual page because of some stupid errors on the github side.Really frustrated by this because it feels like I have been intentionally locked out from commenting on these accessibility issues.

[nohwnd]

I am sorry to hear that. Does it happen only on this repo?

[duaneking]

        Only repro’s owned by Microsoft, as I can otherwise login to github and use it fine and that is why I feel like this is intentional.If Microsoft owns the repo that I have filed a bug on, I cannot see the bug and I cannot reply to it, as every attempt to do so redirects me to an Oauth provider that then locks me out with an error message that sometimes contains a stack trace saying the credentials are not found. Other times it just kicks me out entirely and asks me to login to an internal v- Microsoft account I don’t have access to as I have left ms research.

[nohwnd]

Reported your problem.

[jeffwilcox]

@duaneking: SSO is a pain for sure when changes happen. Can you confirm for me, whether you are still a vendor, and continuing to work, or that you have fully left, and are looking to continue having access?

We have systems that will eventually remove your account from the orgs and fix this issue automatically, but I can help get this fixed sooner.

If you would like to discuss in private, you can email opensource at microsoft.com and we can go into the specifics?

[duaneking]

It seems to be fixed. Thank you for your offers of assistance.

[duaneking]

To be clear: My goal here is no Unicode or anything that would confuse a voice encoder or a standard super cheap braille terminal (under a grand) in the output as blind accessibility requires this stay ASCII text. I requested this be reopened partly out of worry as I was not able to see the contents of the bug, so please ignore that request.

[nohwnd]

At the moment the output contains just letters, basic punctuation characters and parentheses. Success or failure are indicated by the words Failed, Passed, and Skipped.

[jpribele]

When will this make its way into Visual Studio ?

[nohwnd]

in 16.8.0 release. hopefully preview1, I can't say the exact date though.