Testfx: TestContext.TestName in data-driven tests changed behavior in 2.1.2

[majastrz]

Description

After upgrade to MSTest.TestAdapter version 2.2.1 from 2.1.2, our [DataTestMethod] tests changed behavior. TestContext.TestName now contains a badly serialized representation of the data row being passed in.

We are using test names to construct result file paths. This change breaks them on Windows due to Windows file system rules. The package didn't have a major version rev, so this behavior change is unexpected.

If this one intentional, how do we get the old behavior?

Steps to reproduce

What steps can reproduce the defect?

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;

namespace TestProject
{
    [TestClass]
    public class UnitTest1
    {
        public TestContext TestContext { get; set; }

        [TestMethod]
        public void TestMethod1()
        {
        }

        [DataTestMethod]
        [DataRow("hello", "there")]
        [DataRow("general", "kenobi")]
        public void Foo(string one, string two)
        {
            Assert.Fail($"{this.TestContext.TestName} - {one} - {two}");
        }
    }
}

Expected behavior

On 2.1.2, we see the following output from dotnet test:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed Foo [36 ms]
  Failed Foo (hello,there) [33 ms]
  Error Message:
   Assert.Fail failed. Foo - hello - there
  Stack Trace:
     at TestProject.UnitTest1.Foo(String one, String two) in C:\Users\majastrz\source\repos\SerializedGarbage\TestProject\UnitTest1.cs:line 21

  Failed Foo (general,kenobi) [< 1 ms]
  Error Message:
   Assert.Fail failed. Foo - general - kenobi
  Stack Trace:
     at TestProject.UnitTest1.Foo(String one, String two) in C:\Users\majastrz\source\repos\SerializedGarbage\TestProject\UnitTest1.cs:line 21


Failed!  - Failed:     2, Passed:     1, Skipped:     0, Total:     3, Duration: 37 ms - TestProject.dll (netcoreapp3.1)

The relevant part is this: Assert.Fail failed. Foo - hello - there

Actual behavior

On 2.2.1, we see the following output from dotnet test:

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed Foo [100 ms]
  Failed Foo (hello,there) [97 ms]
  Error Message:
   Assert.Fail failed. Foo(System.String,System.String) - hello - there
  Stack Trace:
     at TestProject.UnitTest1.Foo(String one, String two) in C:\Users\majastrz\source\repos\SerializedGarbage\TestProject\UnitTest1.cs:line 21

  Failed Foo (general,kenobi) [< 1 ms]
  Error Message:
   Assert.Fail failed. Foo(System.String,System.String) - general - kenobi
  Stack Trace:
     at TestProject.UnitTest1.Foo(String one, String two) in C:\Users\majastrz\source\repos\SerializedGarbage\TestProject\UnitTest1.cs:line 21


Failed!  - Failed:     2, Passed:     1, Skipped:     0, Total:     3, Duration: 101 ms - TestProject.dll (netcoreapp3.1)

The relevant part is this: Assert.Fail failed. Foo(System.String,System.String) - hello - there

Environment

Repro project file:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.9.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
    <PackageReference Include="coverlet.collector" Version="3.0.3">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

</Project>

The behavior in VS after the upgrade is also strange. TestExplorer chokes on the project after upgrade and refuses to run tests. However, this is reproducible via dotnet test outside of VS as well.

AB#1290016

[majastrz]

Linked the dependabot PR that led to the discovery of this: https://github.com/Azure/bicep/pull/1673

[sumitkm]

@majastrz are you able to run any tests with [DataTestMethod] from Visual Studio? I can't even run the tests from Visual Studio... VS simply creates a new Branch of Tests not executed. The only message is

No test matches the given testcase filter `FullyQualifiedName=Tests.Unit.Common.Utilities.MyTestClass.TestMethod1`

I have tried the clean, restart vs, rebuild to no avail. Going to try and run this from command line and see if is any good.

Someone let me know if I should file this as a separate bug or is this related to the one filed by @majastrz

[majastrz]

Yeah on a new project I wasn't able to run any tests with [DataTestMethod]. Downgrading MSTest.TestAdapter fixed it. On an old project, they somehow worked but had the issue mentioned above.

I think this repo has different rules for submitting bugs in VS. I'd like to keep this issue specific to the TestName behavior change.

[Haplois]

Thank you for the report. I have determined the root cause of the issue. A preview package with the fix will be available later today or tomorrow morning. When it's available, I'll update the issue.

[Haplois]

Two more cases of this same issue: https://github.com/microsoft/testfx/issues/789#issuecomment-793565132, https://github.com/microsoft/testfx/issues/789#issuecomment-792730365

[Haplois]

This is fixed in 2.2.2-preview-20210310-02. The preview package can be found here: https://dev.azure.com/dnceng/public/_packaging?_a=feed&feed=test-tools

[Haplois]

The version v2.2.2 release with the fix.