Vscode: Test Runner Viewlet with an Extension API to plug into [Enhancement Request]

[cloudRoutine]

In response to Erich Gamma's tweet @Krzysztof-Cieslak

A test runner viewlet/sidebar with a pluggable API would be a great addition to vscode.

The viewlet should include visual indications of

• the progress of completed tests out of the set of tests being run
• visual indicators or icons showing basic tests states at the least
  not run | success | failure | ignored/skipped

The test runner should be able to provide a set of sorting categories and repopulate the viewlet based on the users selection

A query box with the ability to filter the tests based on the names shown in the runner's list at a minimum, with the ability for the underlying runner to populate a set of query filters.

Instead of the VS style of collapsing categories

I think the vscode style used in the debugger viewlet is much nicer :smile:

A button could be used to select which test runner you want to use out of the test runner extensions that have been installed

But a dropdown menu would work too

A side button to open a menu could enable the active test runner to populate it with any additional commands specific to its functionality

Some other functionality points -

• When an individual test is selected the result could be rendered in the output panel
• An icon/button to run all tests
• The ability to highlight/select a set of tests and only run those tests
• A basic set of run options, which individual runners might be able to extend
  
• Tests outputs can contain links to specific points in source code
• Debugging specific tests (would be awesome, but understandable if this is too difficult for an initial implementation)
• Associate test runner extensions with specific languages?
• Test playlists (would be nice, but not especially important)
• Ability to specify a task/trigger that will repopulate the viewlet's list of tests
• Dropdown menu that the test runner can population with platform/environment/runtime configurations to use during test execution.

[Krzysztof-Cieslak]

Few ideas, all just IMHO, may be totally wrong ;)

---

I believe that test adapters should have their separate extension point just like debuggers.

Also creating nice, lightweight UI (as described above) would be really great, there is enough place on left panel for one more button ;)

---

From the API point of view I think we would need:

1. Function which returns list of tests for current workspace
2. Define when the test list is repopulated (for some frameworks it can be really dynamic, other requires file saving or even recompilation... all depending on tool and language)
3. Function which takes list of tests and returns their results
4. I guess result should have some kind of standard category - Passed, Failed, Not Run, Ignored

Buttons to run all tests / selected tests / single test / previously not passed tests etc. would all execute function 3. just with appropriate list of tests based on chosen command, selection in UI, and previous result category.

---

A basic set of run options, which individual runners might be able to extend

Basic run options could be easily handled by VSCode (as I've described earlier). Getting API for providing custom options / list could be nice but it's definitely not necessary for initial implementation

Debugging specific tests (would be awesome, but understandable if this is too difficult for an initial implementation)

It's possible to execute start debugging using vscode.startDebug it's just matter of defining which debugger should be started and with what configuration

Open question, should it be defined by test adapter creator or by end user?

Associate test runner extensions with specific languages?

I'd say so - probably just for sake of lazy extension activation. If language supported by test adapter appears in current workspace active test runner.

Ability to specify a task/trigger that will repopulate the viewlet's list of tests

That's actually really interesting case, for most APIs right now VSCode decide when to recalculate things and call implemented API again.

Dropdown menu that the test runner can population with platform/environment/runtime configurations to use during test execution.

I'd like to see something similar to debug case where we have dropdown to select different debugging configurations - different configs specified in launch.json (in case of debuggers, we could have similar file for test adapters)

[et1975]

As an alternative, I find TestDriven's unit-testing experience much better than that of built-in VS or as offered by R#. Personally, I don't need a separate test browser, nor a new output window. I'm already looking at the code and if the plugin can receive which line the cursor is on then it can tell wether I want to run the whole suite or the specific test. For everything else there's always a task and since it can also write to the output window the transition is seamless.

[egamma]

Thanks for the input and it is great to see the different perspectives.

First, test running is an inner development loop activity and is important and fits into Code. However, I was hoping for something more lightweight than a full test explorer. Many test runners like mocha work fine on the command line in the console. Also many of them support continuous test running by watching the files. Something we would not want to replicate.

Rather than starting with the maximum number of features that could be added say to a test explorer, what would be the minimum features to still provide a good developer experience. e.g., navigate from failure traces to the code, seeing an indication of the number of the test failures.

Consider the test runners you are using today, would a command line version for them be sufficient?

[cloudRoutine]

@egamma I wanted to make a strong opening bid :wink:
At a minimum level - navigation to code, the ability to select a subset of tests to run, simple text match based filtering, and showing which tests succeeded, failed, and weren't run.

[Krzysztof-Cieslak]

I guess I would like to see following set of feature:

1. Having some kind of list of tests in workspace and navigating to them
2. Ability to run selected test / set of tests / failed-previously etc.
3. Navigation from error trace to code
4. Having some indicator how many tests are detected, how many passed in previous run, how many failed etc.

Also, having test viewlet, wouldn't break current command line based workflows, so that's just added value :wink:

[egamma]

Just adding how some extensions are now supporting unit testing.

@alexandrudima fyi

Python

Go

• Test at cursor
  
  
  
  • If the cursor is inside a test function, then the test function is run
  
  
  • Output window opens and test results are shown in a channel “GoTests”
  
  
• Test File
  
  
  
  • All the test functions in the file open in active editor are run
  
  
  • Output window opens and test results are shown in a channel “GoTests”
  
  
• Test Package
  
  
  
  • All the tests in the package to which the file open in active editor belongs are run
  
  
  • Output window opens and test results are shown in a channel “GoTests”
  
  
• Test Previous
  
  
  
  • Among the previous 3 commands, whichever was last run is run again
  
  
  • Output window opens and test results are shown in a channel “GoTests”
  
  
• Generate test skeleton at cursor/file/package
  
  
  
  • Creates test file (hello_test.go for hello.go) if not exists
  
  
  • Updates the said test file with test skeletons using gotests for the function at cursor/ all function is the file/ all functions in the package
  
  
• Test coverage in current package

[cloudRoutine]

@egamma if several extensions are starting to implement a common set of features to provide functionality that users clearly want, all of which falls within a set of reoccurring abstract constructs, it seems like the perfect opportunity for vscode to provide a standard API to facilitate this so every language extension doesn't need to roll a bespoke implementation 😸

[egamma]

@cloudRoutine yes, this is exactly the plan.

[samuelkdavis]

Any update on this..?

[matthew-horrocks]

It would be useful if the .runsettings file functionality could be included.

[ghost]

A extension that can access the Activity Bar could speed up the initial implementation by the community.

Is it possible for extension to add activity bar items? i wasn't able to find it...

https://github.com/Microsoft/vscode-extension-samples/blob/master/tree-explorer-sample/USAGE.md
This new API could be maybe helpful

[rmunn]

This is the feature that would be the biggest improvement for me in my VS Code experience. Right now I'm running tests with a command-line runner and manually finding the failing test by a Ctrl+F search on the test names. Having a way to jump to the failing test, and (as a bonus) to see the console output of that specific test in the test output window, would improve my VS Code experience quite a bit.

So I'll echo the "Any update on this?" request from last December. This is my most-wished-for feature in VS Code right now, and I think testing is worthy of its own dedicated icon on the Activity Bar, with shortcut Ctrl/Cmd + Shift + T. (Or fallback shortcut Ctrl/Cmd + Shift + U, for "Unit testing", if Ctrl/Cmd + Shift + T is already taken).

[forki]

Please please please

[vasily-kirichenko]

The VS test runner is the most awful thing I've ever seen. Please don't even look at it for ideas. Look at Rider's (or R#) test explorer and session runner instead, it's a piece of art, really.

[Bomret]

It's almost a year now. Is there any news on this? Maybe just some comment like "It's on the roadmap for 201x" or "No, we don't want to implement this now, because..."?

[realvictorprm]

@vasily-kirichenko What about posting some screenshots here so it's easier to follow your suggestion.

[vasily-kirichenko]

@realvictorprm are u gonna implement it? if so, install rider and play with the test runner.

[felix-b]

Based on lack of any updates from the maintainers, it feels like the community has to drive it, if we want to have it any time soon.

Obviously enough, we want a common infrastructure in VSCode, and adapters to different platforms/tools/frameworks.

This is quite a big task though. I'd prefer moving step by step, and not trying to provide a complete solution right off.

I personally need integrated unit testing experience for .NET Core C# projects, so I'm thinking of these tactical steps:

• Implement a monolith C# test runner extension for VSCode, integrated with OmniSharp. Learn the lessons.
• Once C# test runner works, it can be a good starting point for design and extraction of common test runner API. Here JavaScript in browser/Node and other platforms/frameworks need to be taken into account.
• Transition the monolith C# test runner into the defined architecture. Learn/adjust as necessary.
• Propose a PR to VSCode for the common test adapter infrastructure and API.

[egamma]

This feature is still on our roadmap and we want to better support test running to complete the inner loop of development.

Today, there are already many extensions that support test running. Our plan is to start by analyzing these extensions and then to come up with APIs and UIs that unifies the experience for the user and the test runner implementors.

It is too early to tell whether this will be a general test adapter API or just a smaller building block like a TestFailure. A TestFailure represents the result from a test runner and this would be similar to a Diagnostic which represents errors/warnings from a compiler. Having a TestFailures abstraction allows us to render test results in a common way in VS Code, e.g. in a Test Results view or the status bar can show the current number of failed tests.

[samuelkdavis]

Please don't rely too much on the existing testing extensions - they're a poor mans set of test runners. Check out ReSharpers support for unit testing. The lack of a test explorer leaves a painful hole when coding with vscode https://www.jetbrains.com/resharper/features/unit_testing.html

[xsoheilalizadeh]

.NET Core Test Explorer for Visual Studio Code

[gfrancischini]

A work in progress test extension is published here: https://github.com/gfrancischini/vscode-unit-test

The idea is to reuse the vstest framework to be able to run any test framework that run inside Visual Studio Test Explorer.

[rmunn]

In https://github.com/Microsoft/vscode/issues/9505#issuecomment-311880580, @egamma wrote:

This feature is still on our roadmap and we want to better support test running to complete the inner loop of development.

However, it no longer appears on the 2018 roadmap. Has it been dropped from consideration? Should extensions start implementing their own ad-hoc test views, rather than wait for VS Code to provide this? Or is it still on the roadmap, but the wiki just hasn't been brought up-to-date yet?

[egamma]

@rmunn we did an investigation of the existing Test Running extensions during the last planning period (we should have updated the issue with the findings sorry). We have discovered a lot of creativity with regard to Test Running extensions. After this investigation we came to the conclusion that trying to come up with some basic Test Running infrastructure in the VS Code core would not add a lot of value and could be more constraining. Finally our users seem to be happy with the current test extensions. For this reason we did not add it to the 2018 roadmap. This doesn´t mean that it will never come back and we will continue watching and listening in this space.

[DanTup]

This is kinda disappointing for two reasons:

1. There will be no consistency for test running across languages (something that VS Code seems to have worked hard to maintain for other functionality)
2. There's no nice existing place to render tests in which means doing it ourselves we have to pick the best compromise (eg. between putting tests into the explorer pane, the output pane, rendering a custom html view in a "file" tab)

I know Code doesn't want to be an IDE, but tests seems like one of the most important things that most projects don't do well (software complexity is only increasing and quality seems to be decreasing). Having good built-in test functionality could make a real difference.

[uxsoft]

Hi, we need this!

The existing extensions are a desperation attempt to fix something you should have fixed a long time ago. There needs to be a unified place where I can see all the tests from different plugins for test frameworks. Then a test framework maintainer can just provide a plugin and the test Tab could show an aggregation. Now there is no place where I can see both .net and js tests at the same place. We are definitely not happy with them. This feature seems like a no-brainer and not too difficult to engineer.

[rmunn]

If as @egamma points out, there's a lot of variation between the different test runners, then it might make some sense to hold off on implementing this API for a while.

However, I'd like to request one feature be implemented now, at least: adding a new sidebar view (and icon) for unit testing extensions to use. Currently the API only allows extensions to add tree views to the explorer or debug sidebars, which means that unit test extensions like https://github.com/gfrancischini/vscode-unit-test have to either use the debug sidebar (which doesn't feel appropriate) or the explorer sidebar (which is already overloaded, with three panels — open editors, folder view, and bookmarks — opened by default, and that's not counting any other panels that other extensions might add). I'd much rather see a sidebar view, with its own icon, made available for unit test extensions to use. Here's a mockup of what it could look like:

The icon could, of course, be anything: I just picked something that seemed appropriate (in this case, an icon from https://icons8.com/icon/6258/checked with a free license).

As someone else has already mentioned in the thread, unit testing is one of the things that feels fundamental to the concept of "a text editor for programmers". I know you want to avoid sidebar-icon proliferation, but unit testing is one of the things that's so core to the coding experience that I believe it truly deserves its own sidebar icon.

General question: would you prefer me to open a new issue to track this "new sidebar icon for unit testing" idea, or should it be tracked here as a subset of this issue?

[rmunn]

https://github.com/Microsoft/vscode/issues/43645 appears to be related to my request for a new sidebar icon for test runner extensions to use.

[uxsoft]

Sounds like a good start

[DanTup]

If as @egamma points out, there's a lot of variation between the different test runners, then it might make some sense to hold off on implementing this API for a while.

Let's not confuse variation with inconsistency. Currently people are left to implement it themselves with no guidelines, so of course they'll all be different.

Visual Studio has a Test window which, while not perfect, has served many languages and frameworks reasonably well. VS Code has a good process for suggesting (and later implementing) proposed APIs to get feedback. I don't see why this wouldn't work well here. Propose something, and let all the people that want this (of which there are many) provide feedback if it won't fit their needs.

[hbenl]

FYI: I have just published a Test Explorer extension that is trying to fill this gap. This extension can be used by other extensions (called Test Adapters) to show and run the tests for a particular testing framework. I have also published a Test Adapter extension for Mocha and I'm working on another one for Jasmine.
The feature set of these extensions is currently fairly basic but I intend to add more features in the coming months and hope that the community will add more adapters.

[luabud]

I guess I would like to see following set of feature:

1. Having some kind of list of tests in workspace and navigating to them
2. Ability to run selected test / set of tests / failed-previously etc.
3. Navigation from error trace to code
4. Having some indicator how many tests are detected, how many passed in previous run, how many failed etc.

Agreed, and I also find useful to have a "show output" option.

[Kradirhamik]

Will there be more news on this please?

[connor4312]

We are starting to investigate this here 👉 https://github.com/microsoft/vscode/issues/107467