Xunit: dotnet-test-xunit fails to discover tests for net451 when we pass --designtime and --list

[livarcocc]

<root>\cli\test\dotnet-test.Tests\bin\Debug\netcoreapp1.0\It_runs_tests_for_net451\ProjectsWithTests\MultipleFrameworkProject\bin\Debug\net451
\win10-x64\dotnet-test-xunit.exe <root>\cli\test\dotnet-test.Tests\bin\Debug\netcoreapp1.0\It_r
uns_tests_for_net451\ProjectsWithTests\MultipleFrameworkProject\bin\Debug\net451\win10-x64\MultipleFrameworkProject.dll
--designtime --list

The key here are the designtime and list parameters. This will cause the runner to try to obtain source information when discovering the tests.

With these parameters, the discoverysink will have no tests in it. If you omit the designtime (which will cause us to not try to get source information), it will find the right tests.

I checked and I saw that filename and filenumbers were being returned by the source information provider. I haven't debugged inside the controller yet. This will be my next step.

This is the project.json for the project:

{
  "version": "1.0.0-*",
  "dependencies": {
    "dotnet-test-xunit": "2.2.0-preview3-build1032",
    "Microsoft.NETCore.Platforms": "1.0.1",
    "xunit": "2.2.0-beta3-build3330"
  },
  "frameworks": {
    "netcoreapp1.0": {
      "imports": [
        "dotnet",
        "portable-dnxcore50+net45+win8+wp8+wpa81",
        "portable-net45+win8"
      ],
      "dependencies": {
        "Microsoft.NETCore.App": {
          "type": "platform",
          "version": "1.0.0"
        },
        "System.Linq.Expressions": "4.1.0",
        "System.Runtime.Serialization.Primitives": "4.1.1"
      }
    },
    "net451": {}
  },
  "testRunner": "xunit"
}

cc @bradwilson @piotrpMSFT

[sstublic]

Is it possible to get any ETA on this please? It prevents us to use goodies from new versions of dotnet-test-xunit in VS if we target net451.

[sstublic]

Ok, I think I narrowed the problem down. I've managed to create a package that works for me just fine, but the solution I used is too dirty for PR and I still don't understand whats going on.

There were two problems I found.

One was that SourceInformation class was not serializable and that was throwing exception in NET451. SerializableAttribute is not available in .NET Core AFAIK so this was easy to narrow down and solve. I just added conditional attribute to the class
https://github.com/xunit/dotnet-test-xunit/blob/master/src/dotnet-test-xunit/TestHost/SourceInformationProvider.cs#L37

#if NET451
        [Serializable]
#endif
        class SourceInformation : ISourceInformation

Second problem was completely incomprehensible to me and very hard to track down.

DesignTimeExecutionSink.cs (https://github.com/xunit/dotnet-test-xunit/blob/master/src/dotnet-test-xunit/TestHost/DesignTimeExecutionSink.cs) uses a Dictionary<> to convert the TestCases from xUnit object to Vs objects. That lookup always fails.
The funny thing is that it fails while there is a key with same hashcode in the Dictionary. That key will also yield true to '==' comparison and to .Equals() comparison with the lookup key, but the lookup will still fail.
I'm guessing it has something to the with Remoting objects and availability of default IEqualityComparer, but beyond that I'm clueless.

When I patched up those dictionary lookups with conversions.Where(pair => pair.Key == testCase).Select(pair => pair.Value).Single(); instead of conversions[testCase] everything started working fine.

@bradwilson All provided tests are passing, however this solution is not good enough. I hope my findings will help you come up with correct solution to this issue.

EDIT: and now I finally have my outputs for passing tests as well. YEY ! :)

[kwaclaw]

@sstublic Following your comments about the failed dictionary lookup, and based on the assumption (which may be wrong!) that ITestCase instances should be compared only based on their UniqueID member, I defined a class called "TestCaseEqualityComparer" like this:

    public class TestCaseEqualityComparer: IEqualityComparer<ITestCase> {
        static TestCaseEqualityComparer() {
            Default = new TestCaseEqualityComparer();
        }

        public static TestCaseEqualityComparer Default { get; private set; }

        public bool Equals(ITestCase x, ITestCase y) {
            if (object.ReferenceEquals(x, y))
                return true;
            if (x == null || y == null)
                return false;
            return string.Equals(x.UniqueID, y.UniqueID);
        }

        public int GetHashCode(ITestCase obj) {
            return obj?.UniqueID?.GetHashCode() ?? 0;
        }
    }

and then supplied its Default instance to the dictionary constructor in DesignTimeTestConverter.cs:

...
            var results = new Dictionary<ITestCase, VsTestCase>(TestCaseEqualityComparer.Default);
            foreach (var group in groups)
...

This seems to work - my test cases are discovered.

I would think that the TestCaseEqualityComparer should be part of xUnit itself rather than dotnet-test-xunit.

Karl

[bradwilson]

Closed for age.