Testfx: Provide a way to opt-out of discovery time expanding of ITestDataSource tests

[Haplois]

Description

We recently changed how ITestDataSource tests are discovered. We need to provide a way to opt-out from this behavior.

Proposal

• Ability to opt-out assembly, type or test case level by using an attribute.
• Ability to opt-out using a command-line parameter or .runsettings setting.

Steps to reproduce

protected static readonly Guid _subscriptionId = Guid.NewGuid();

[DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
[DataTestMethod]
public void MyTest1(Guid subscriptionId)
{
}

private static IEnumerable<object[]> GetData()
{
    yield return new object[]
    {
        _subscriptionId
    };
}

Expected behavior

Discovered Tests: 
 - MyTest1

Actual behavior

Discovered Tests:
 - MyTest1 (e6a883d7-03b5-4231-a06f-99e5847eb9b0)

AB#1353020

[patest-dev]

The opt-out is very(!!) much needed and all this new discovery time behavior should actually be an opt-in.
How could all of these breaking changes be introduced on a minor version??

Even a DynamicData test as simple as this one, is now only being executed 2-times and the null case is simply lost

private static IEnumerable<object[]> GetData()
{
    List<object[]> ret = new List<object[]>()
    {
        new object[] { null },
        new object[] { "" },
        new object[] { "a" },
    };
    return ret;
}

[TestMethod]
[DynamicData(nameof(GetData), DynamicDataSourceType.Method)]
public void StringTest(string test)
{
    Console.WriteLine(test);
}

[Haplois]

Support is added by https://github.com/microsoft/testfx/commit/9a621beab927612b357c3f68f97ead58c6282864 to version 2.2.6. Currently we only support assembly level configuration via an attribute.

.testsettings file and class & method level configuration is not possible yet. Depending on feedback, we can add it.

[abatishchev]

@Haplois thank you! can you please document this and provide a timeline when the nuget package will be available on nuget.org?

[Haplois]

@abatishchev assuming all the tests pass, I'll release it today.

[abatishchev]

@Haplois what should be the package versions in order to consume the new functionality?

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.6" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.6" />

Like this?

[Haplois]

@Haplois what should be the package versions in order to consume the new functionality?

<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.11.0" />
<PackageReference Include="MSTest.TestAdapter" Version="2.2.6" />
<PackageReference Include="MSTest.TestFramework" Version="2.2.6" />

Like this?

Yes. That's correct. You also need to define TestDataSourceDiscoveryAttribute on assembly level like:

[assembly: TestDataSourceDiscovery(TestDataSourceDiscoveryOption.DuringExecution)]