Vstest: Discover tests in parallel

[JamesNK]

Description

It takes 30 seconds to discover all the ASP.NET MVC tests, mostly because there are multiple test projects. Discovering tests in parallel across projects would speed this process up.

Steps to reproduce

1. Clone https://github.com/aspnet/mvc
2. Open sln in VS
3. Wait 30 seconds for tests to be discovered in projects one at a time

Expected behavior

VS should discover tests in multiple projects at the same time.

Actual behavior

Tests are discovered one project at a time.

Diagnostic logs

[9/07/2018 1:34:56 PM Informational] ------ Discover test started ------
[9/07/2018 1:34:57 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (32-bit Universal Windows)
[9/07/2018 1:34:57 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (32-bit Desktop .NET 4.0.30319.42000)
[9/07/2018 1:34:59 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:34:59 PM Informational] [xUnit.net 00:00:00.42]   Discovering: Microsoft.AspNetCore.Mvc.Abstractions.Test
[9/07/2018 1:35:00 PM Informational] [xUnit.net 00:00:00.60]   Discovered:  Microsoft.AspNetCore.Mvc.Abstractions.Test
[9/07/2018 1:35:00 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:01 PM Informational] [xUnit.net 00:00:00.70]   Discovering: Microsoft.AspNetCore.Mvc.Analyzers.Experimental.Test
[9/07/2018 1:35:01 PM Informational] [xUnit.net 00:00:00.81]   Discovered:  Microsoft.AspNetCore.Mvc.Analyzers.Experimental.Test
[9/07/2018 1:35:01 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:02 PM Informational] [xUnit.net 00:00:00.41]   Discovering: Microsoft.AspNetCore.Mvc.ApiExplorer.Test
[9/07/2018 1:35:02 PM Informational] [xUnit.net 00:00:00.57]   Discovered:  Microsoft.AspNetCore.Mvc.ApiExplorer.Test
[9/07/2018 1:35:02 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:03 PM Informational] [xUnit.net 00:00:00.50]   Discovering: Microsoft.AspNetCore.Mvc.Core.TestCommon
[9/07/2018 1:35:03 PM Informational] [xUnit.net 00:00:00.53]   Discovered:  Microsoft.AspNetCore.Mvc.Core.TestCommon
[9/07/2018 1:35:04 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:04 PM Informational] [xUnit.net 00:00:00.58]   Discovering: Microsoft.AspNetCore.Mvc.Core.Test
[9/07/2018 1:35:06 PM Informational] [xUnit.net 00:00:02.37]   Discovered:  Microsoft.AspNetCore.Mvc.Core.Test
[9/07/2018 1:35:07 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:07 PM Informational] [xUnit.net 00:00:00.48]   Discovering: Microsoft.AspNetCore.Mvc.Cors.Test
[9/07/2018 1:35:07 PM Informational] [xUnit.net 00:00:00.62]   Discovered:  Microsoft.AspNetCore.Mvc.Cors.Test
[9/07/2018 1:35:08 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:08 PM Informational] [xUnit.net 00:00:00.46]   Discovering: Microsoft.AspNetCore.Mvc.DataAnnotations.Test
[9/07/2018 1:35:08 PM Informational] [xUnit.net 00:00:00.64]   Discovered:  Microsoft.AspNetCore.Mvc.DataAnnotations.Test
[9/07/2018 1:35:09 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:09 PM Informational] [xUnit.net 00:00:00.43]   Discovering: Microsoft.AspNetCore.Mvc.Formatters.Json.Test
[9/07/2018 1:35:09 PM Informational] [xUnit.net 00:00:00.59]   Discovered:  Microsoft.AspNetCore.Mvc.Formatters.Json.Test
[9/07/2018 1:35:10 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:10 PM Informational] [xUnit.net 00:00:00.45]   Discovering: Microsoft.AspNetCore.Mvc.Formatters.Xml.Test
[9/07/2018 1:35:10 PM Informational] [xUnit.net 00:00:00.65]   Discovered:  Microsoft.AspNetCore.Mvc.Formatters.Xml.Test
[9/07/2018 1:35:11 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:12 PM Informational] [xUnit.net 00:00:00.89]   Discovering: Microsoft.AspNetCore.Mvc.FunctionalTests
[9/07/2018 1:35:12 PM Informational] [xUnit.net 00:00:01.25]   Discovered:  Microsoft.AspNetCore.Mvc.FunctionalTests
[9/07/2018 1:35:13 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:13 PM Informational] [xUnit.net 00:00:00.64]   Discovering: Microsoft.AspNetCore.Mvc.IntegrationTests
[9/07/2018 1:35:14 PM Informational] [xUnit.net 00:00:00.84]   Discovered:  Microsoft.AspNetCore.Mvc.IntegrationTests
[9/07/2018 1:35:14 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:15 PM Informational] [xUnit.net 00:00:00.61]   Discovering: Microsoft.AspNetCore.Mvc.Localization.Test
[9/07/2018 1:35:15 PM Informational] [xUnit.net 00:00:00.76]   Discovered:  Microsoft.AspNetCore.Mvc.Localization.Test
[9/07/2018 1:35:15 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:16 PM Informational] [xUnit.net 00:00:00.64]   Discovering: Microsoft.AspNetCore.Mvc.Razor.Test
[9/07/2018 1:35:16 PM Informational] [xUnit.net 00:00:00.84]   Discovered:  Microsoft.AspNetCore.Mvc.Razor.Test
[9/07/2018 1:35:17 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:17 PM Informational] [xUnit.net 00:00:00.63]   Discovering: Microsoft.AspNetCore.Mvc.RazorPages.Test
[9/07/2018 1:35:17 PM Informational] [xUnit.net 00:00:00.83]   Discovered:  Microsoft.AspNetCore.Mvc.RazorPages.Test
[9/07/2018 1:35:18 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:18 PM Informational] [xUnit.net 00:00:00.61]   Discovering: Microsoft.AspNetCore.Mvc.TagHelpers.Test
[9/07/2018 1:35:19 PM Informational] [xUnit.net 00:00:00.85]   Discovered:  Microsoft.AspNetCore.Mvc.TagHelpers.Test
[9/07/2018 1:35:19 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:20 PM Informational] [xUnit.net 00:00:00.36]   Discovering: Microsoft.AspNetCore.Mvc.TestDiagnosticListener
[9/07/2018 1:35:20 PM Informational] [xUnit.net 00:00:00.37]   Discovered:  Microsoft.AspNetCore.Mvc.TestDiagnosticListener
[9/07/2018 1:35:20 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:21 PM Informational] [xUnit.net 00:00:00.64]   Discovering: Microsoft.AspNetCore.Mvc.Test
[9/07/2018 1:35:21 PM Informational] [xUnit.net 00:00:00.75]   Discovered:  Microsoft.AspNetCore.Mvc.Test
[9/07/2018 1:35:21 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:22 PM Informational] [xUnit.net 00:00:00.46]   Discovering: Microsoft.AspNetCore.Mvc.ViewFeatures.Test
[9/07/2018 1:35:22 PM Informational] [xUnit.net 00:00:00.99]   Discovered:  Microsoft.AspNetCore.Mvc.ViewFeatures.Test
[9/07/2018 1:35:23 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:23 PM Informational] [xUnit.net 00:00:00.44]   Discovering: Microsoft.AspNetCore.Mvc.Views.TestCommon
[9/07/2018 1:35:23 PM Informational] [xUnit.net 00:00:00.46]   Discovered:  Microsoft.AspNetCore.Mvc.Views.TestCommon
[9/07/2018 1:35:24 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:24 PM Informational] [xUnit.net 00:00:00.46]   Discovering: Microsoft.AspNetCore.Mvc.WebApiCompatShimTest
[9/07/2018 1:35:24 PM Informational] [xUnit.net 00:00:00.65]   Discovered:  Microsoft.AspNetCore.Mvc.WebApiCompatShimTest
[9/07/2018 1:35:25 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0-rc.1.build4038 (64-bit .NET Core 4.6.26616.03)
[9/07/2018 1:35:25 PM Informational] [xUnit.net 00:00:00.72]   Discovering: Mvc.Analyzers.Test
[9/07/2018 1:35:26 PM Informational] [xUnit.net 00:00:00.84]   Discovered:  Mvc.Analyzers.Test
[9/07/2018 1:35:26 PM Informational] ========== Discover test finished: 9837 found (0:00:29.6328847) ==========

Environment

Visual Studio 15.7.4

[pvlakshm]

@JamesNK, may i assume that this is a VS IDE-specific "ask"?

[JamesNK]

yes

[pvlakshm]

OK. Thanks.
Tagging @ManishJayaswal.

[ManishJayaswal]

With real time test discovery, tests are discovered as the files get added to Roslyn workspace so it should not be per project. Additionally you should not need a build to discover tests. Can you check you tool-options-tests to see if real time test discovery is on or not?

[JamesNK]

It is turned off because it doesn't discover all tests for me. You can log an issue about that if you want to: MVC solution runs 3833 tests after opening the solution with real time turned on, 10000+ with it turned off.

[smadala]

FYI Related issue https://github.com/Microsoft/vstest/issues/499.

/cc @smalpani-msft

[pvlakshm]

@ManishJayaswal, @JamesNK,
Will wait to see if this is addressed via real-time-test discovery.

[JamesNK]

Even when I turn on real time test discovery, after rebuilding there is still a 30 second wait while VS re-discovers tests. During this time the "Run All" button is disabled and I have to wait.

Have any of you tried out the solution I linked to?

[ManishJayaswal]

After building the regular reflection based discovery will kick in which is taking this time. This is the default behavior. We chose this default because real time discovery may not discover certain types of tests. If real time test discovery is discovering all tests in your solution ( which I don't think is the case based on your earlier comments) then you can go to tools, options, test and disable reflection based discovery after build.
@Shyam-Gupta is taking a look at your solution. He will respond back once he identifies what the issue is. He was mentioning that the solution is using some old preview of .NET Core SDK.

[Shyam-Gupta]

I was able to get past .NET Core SDK problem by setting up path variables appropriately on the machine.
I am able to repro the issue. Looks like MTFM is in use in test projects which could be causing the issue. But I am yet to investigate the root cause. I will get back soon with more data.

[JamesNK]

Look at all the likes on this tweet: https://twitter.com/JamesNK/status/1393132802384613382

This is still an issue.

[Sanan07]

@JamesNK we already have "Run in Parallel" option inside TestExplorer, it also discovers in parallel.
I has some small potential issues, when user aborts discovery and starts execution, but if you don't do that and just wait until all discovery finishes, so it should work fine.

[JamesNK]

@Sanan07 This is NOT about running tests in parallel. This is about discovering tests in multiple projects in parallel.

Please do me a favour and reread this issue form the start.

1. I supplied a log file that clearly shows that test discovery takes a long time when there are many projects. Is 30 seconds to discover 10000 tests acceptable?
2. I supplied a repo of the issue. After denying that is was a problem, I finally convinced someone who works on vstest tried out. They agreed that there is an issue.

I am able to repro the issue. Looks like MTFM is in use in test projects which could be causing the issue. But I am yet to investigate the root cause. I will get back soon with more data.