Sdk: ReSharper support for dotnet-test. Questions and feedback

[citizenmatt]

Hi all. I want to implement support for running dotnet/cli tests in ReSharper. I've got ReSharper working with DNX RC1, but hit a few bumps along the way. The current IDE integration for testing is based on what Visual Studio needs, and ReSharper is somewhat different (e.g. we do in-editor, as-you-type discovery, structure tests hierarchically, etc.). After a chat with some of the team at Build (@piotrpMSFT, @terrajobst, but sadly not @ellism), I’ve got some questions and feedback for RC2.

I’ve followed along with the design of the new model (#1376 and dotnet/cli#2069), but not the implementation, so my understanding could easily be out of date. I've split things into test discovery, execution, debugging and profiling/coverage.

Test discovery

ReSharper does test discovery as you type, by examining the source, but dotnet/cli requires a test ID created by the framework in a separate discovery step. This means ReSharper has to map between its discovered tests, and the dotnet/cli ID. The issue is that the ID is an opaque value, not formalised, and can’t be generated from source (for row tests, xunit returns an ID that includes the hash of the arguments).

Issues:

• ReSharper has to call dnx.exe/dotnet.exe twice - once to discover tests on each build, and again to execute them. This is usually in the same step (build before testing) so can we reuse a single dotnet.exe invocation? Similarly, is there a way to do an up-to-date check to avoid the cost of discovery if the build is already up-to-date? (This is where I don't know enough about RC2's handling of multi-targeting and output files)
• Mapping in RC1 relies on knowledge of the ID format. Adding type and method/member name to the discovery message would remove this knowledge, at least for non-row tests.
• Currently, the in-editor discovery for xunit does not discover theories/row tests. So given a type and method name, we can map to the owning method test container, and create new elements in the test hierarchy based on this, the display name and the test ID.
• However, ReSharper supports in-editor discovery for nunit row tests (e.g. [TestCase("Hello to Jason Isaacs")]), and xunit could, in the future. It would be great to have a way to map both sets of discovered row tests, but I don't know what that would be...

Running tests

• I need to run dotnet.exe - how do I find it? Do I rely on it being in the path?
• DNX RC1 did not report test output, either as it was happening or at the end of the run
• There is currently no way to cancel a test run. Can we have a message that we send that will signal the test runner to end? This can be a co-operative thing - wait until the current test(s) end, rather than having to interrupt a running test (ReSharper will kill processes if you request cancel twice)
• Does the test runner need to tell the IDE about cache folders to clean up if the test run is aborted? E.g. on the desktop, shadow copy creates a temp folder that the ReSharper process will clean up if it kills processes. I know shadow copy isn't supported any more, at least on CoreCLR, but is there a need for this for non-CoreCLR targets?
• dotnet-test.exe should connect to a socket opened by the IDE, rather than the IDE finding a port number and making dotnet-test.exe open it. Based on a conversation with @piotrpMSFT, I think this is already fixed, although the issue (#1949) hasn’t been updated.
• How are errors in the test runner reported to the IDE?

• What is the expected behaviour of the IDE if the version in the protocol isn’t 1?

  Debugging tests

• What’s the correct GUID for debugging a CoreCLR project? I cheated when implementing this for RC1 - I messed around until I found one that worked :)

• How do I know when tests are being run on CoreCLR vs desktop CLR vs Mono, so I can use the correct debugger?
  
  
  
  • DNX RC1 created launchSettings.json from the VS property pages, and there was a series of undocumented fallbacks and defaults. What's the RC2 process?
    
    

    Profiling/code coverage

  
  

We need to inject code into the test runner for profiling and code coverage, so that the profilers know when a test starts/ends.

• How do we tell the test runner to go into single threaded mode (run one tests at a time, and report messages synchronously)? If we’re running in parallel mode, the profiling/coverage stats are all wrong.
• How do we load the client side code into the process?
  
  
  
  • It looks like the current solution is to add a nuget package in project.json, but this won’t work from the UI controls we already have for initiating profiling/coverage. Can this be automated?
  
  
  • The implementation of the current nuget package for VS is a “shim” that has a single method WriteSymbols. When is this called, and by whom?
  
  
• How does this client side code hook into the test pipeline? How does it know what tests are started/finished and when?

Apologies for this being a bit of a brain dump. I can split it into smaller issues if it makes it easier.

[rdefreitas]

@citizenmatt brings us some great points here, and I don't need to tell you that Resharper support is a must have considering just how much of the community has purchased it.

Can we get any update of some of his concerns?

[niemyjski]

Any updates on this.

[TheRealPiotrP]

@citizenmatt is tracking this issue offline with the @dotnet/vstest team. Closing the CLI issue as it is inactive. @citizenmatt if I'm missunderstanding the status please comment and I will help.

[citizenmatt]

Yes, we're going to pick up these issues with the @dotnet/vstest team.