Vstest: Feature Request: dotnet test --failing-only

[TheRealPiotrP]

@blackdwarf maybe you can help flesh out the design

We have some rather large test projects which take 3-5 mins to run. In these projects we'll sometimes make a change that breaks a few tests, creating a terrible F5 loop because we need to re-run the whole project to validate fixes. It would be wonderful if the test console could just re-run tests that previeously failed. One easy implementation would be for trx logs to get produced and then consumed on subsequent runs.

--failing-only is likely not the right option string, but gets the point across.

Thoughts?

/cc @livarcocc

[livarcocc]

I love this idea. It is similar to what you have in the IDE. Maybe we should even come up with something more powerful to control test execution based on different criteria. Like, only passed ones, only failed ones, only the ones that haven't run before, so on and so forth, just like we can in the IDE.

[blackdwarf]

VS already does this, right? Does it do this by parsing out the trx files? I do like the idea, btw.

As for the UX, you could hang it off the filter: dotnet test --filter Failing or similar. Failing becomes a special category of a sorts.

[codito]

VS stores the context across runs in-memory (without any trx parsing). Console runner starts afresh each time. It will require a way to find which tests failed in the last run, it also needs to understand whether user ran dotnet test on this csproj or another csproj in last run.

May be this can be supported via some sort of scripting?

cat foo.trx | grep Failed | dotnet test --filter

[blackdwarf]

@codito so, this information is contained within the TRX file?

Is your concern that this could impact performance, because the TRX file could be big?

[codito]

@blackdwarf yes, the failed tests information is in a trx file. It sounds little odd for a core feature to depend on an extension though. E.g. the user may not use trx logger, or prefers another logger.

Second, finding the right context for last test run will be challenging. Few options that come to mind: implement special logger that keeps data in ~/.vstest, or ask user for last log file. In IDE, this is an easy problem since boundary of context is well defined: start saving testcase/result on sln load and erase on sln close :)

Why not offload the responsibility of finding failed tests to other tools in command line?

[TheRealPiotrP]

@codito so dotnet test instead of vstestconsole?

[blackdwarf]

@codito to elaborate somewhat on what @piotrpMSFT said above. We could say that the solution here is to use something else on the command line to parse out the logging file. This, however, is not an optimal solution for several reasons, the primary one being that it is platform-specific; the commands you use in bash or similar UNIX shell will not translate to cmd or powershell on Windows. This kind of defeats

Another solution that presents itself here would be to add this functionality in dotnet test in the following way:

1. Upon a run, we automatically log the failed tests to a file.
2. When the user invokes dotnet test --filter Faling or dotnet test --failing or something like that, dotnet test implementation checks for the existence of the file and uses it.
3. The command passes a specially crafted vstestconsole invocation "downards".

The file can contain only the last run for an MVP to be tested out.

Thoughts on this approach?

[codito]

+1 on this approach. This file could be a state or lastrun file. It may be tied to the project, similar to states saved by other tools (project.assets.json in case of restore for instance) and is decoupled from the actual tool invocation.

[blackdwarf]

@codito for sure and it should live in the obj folder as it is easily generated via normal development activity. 😄

[pvlakshm]

In general the console runner neither remembers nor refers-to state from a previous run. While I understand the need to be able to run only failing tests, that is a responsibility that needs to be borne by a tool/script outside of the vstest console runner.

[lauxjpn]

I use the following PowerShell command to run only the previously failed tests again, based on the newest trx file in .\TestResults\:

dotnet vstest '.\bin\Debug\netcoreapp3.0\MyTests.dll' /Logger:trx /Tests:"$((Select-Xml -Path (gci '.\TestResults\' | sort LastWriteTime | select -last 1).FullName -XPath "//ns:UnitTestResult[@outcome='Failed']/@testName" -Namespace @{"ns"="http://microsoft.com/schemas/VisualStudio/TeamTest/2010"}).Node.Value | % {$_ -replace '^My\.Long\.And\.Tedious\.Namespace\.', ''} | % {$_ -replace '^(.*?)\(.*$','$1'} | Join-String -Separator ','))"

Beware, that there is a character limit on the maximum command line length, that can easily be hit when many tests have previously failed.
Use the % {$_ -replace '^My\.Long\.And\.Tedious\.Namespace\.', ''} part, to get rid of namespace prefixes, if you can.