Nsubstitute: It can't get return object in debug mode once I customize code coverage analysis

[MilesLin]

Describe the bug
I customized code coverage via a sample from Customize code coverage analysis. After that, the object always returns null when I set a breakpoint in debug mode.

And I found the critical part is an attribute called friendlyName="Code Coverage" in CodeCoverage.runsettings. If you set friendlyName="Code Coverage" a space between Code and Coverage the bug show up whereas set friendlyName="CodeCoverage" no space between Code and Coverage the bug disappear.

To Reproduce
I created a sample on NSub_Bug_Report.
Please load CodeCoverage.runsettings before running the test.

1. Set breakpoint at GoBug() method in Class1.cs
2. Run Debug test GoBug_StateUnderTest_ExpectedBehavior() in Class1Tests.cs
3. To see employee obj in debug mode.

Expected behaviour
To see the return object when loading CodeCoverage.runsettings.

Environment:

• NSubstitute version: 3.1.0
• Platform:
  
  
  
  • doenetcore2.1 project on windows
  
  
  • Microsoft Visual Studio Enterprise 2017 Version 15.9.2
  
  

[alexandrnikitin]

Hi @MilesLin

Thank you for the awesome repro project! It helps a lot.
I just wanted to note. I tried to reproduce the issue and it works for me. I used VS runner for that via Test -> Debug -> All Tests. The thing I noticed is that it prints the following in the "Tests" output window:

[12/21/2018 11:17:38 PM Informational] ------ Run test started ------
[12/21/2018 11:17:38 PM Warning] Code coverage is not supported while debugging tests. Code coverage has been disabled for this debug session.
[12/21/2018 11:17:42 PM Informational] [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.0 (64-bit .NET Core 4.6.27019.06)
[12/21/2018 11:17:43 PM Informational] [xUnit.net 00:00:01.60]   Discovering: MyLib.UnitTests

As I see Code coverage and debugging don't play well together. I can assume that you use Resharper test runner, don't you? It's possible that Resharper doesn't disable test coverage when in debug mode. I don't it to check unfortunately.

[MilesLin]

HI @alexandrnikitin

Thank you for the quick response. Unfortunately, I don't install Resharper. And I also see the same message as yours in "Tests" output window.
I made a record on Youtube to what I faced NSub Bug Report With CodeCoverage 2018/12/22, hope this will help you.

And this issue also happened on my colleague's computer.

If you any details, please let me know.

Or maybe this is the bug related to Microsoft?

[alexandrnikitin]

Hmm, interesting, I can't reproduce the bug. I have the same versions as you do. dotnet 2.1.502 and VS Community 15.9.4. It works for me. I doubt that it's an NSubstitute bug.

[dtchepak]

Hi @MilesLin,

If you change the test to use a real Employee instance instead of a substitute does the problem still occur? Something like:

        class TestEmployee : IEmployee {
            private Employee employee;
            public TestEmployee(Employee e) { employee = e; }
            public Employee Get() { return employee; }
        }

        [Fact]
        public void GoBug_StateUnderTest_ExpectedBehavior()
        {
            // Arrange
            var subEmployee = new TestEmployee(new Employee()
            {
                Id = 1,
                Name = "Bob"
            });

            var unitUnderTest = new Class1(subEmployee);

            // Act
            unitUnderTest.GoBug();

            // Assert
            Assert.True(false);
        }

That might help to work out whether it is related to NSubstitute.

It might also be worthwhile making sure the project is cleaned/rebuilt each time to make sure debug symbols are updated properly between test runs.

[MilesLin]

@dtchepak
It is the same whether I use NSub or TestEmployee. Thank you for helping me to clarify where the bug belongs to. I will report to Microsoft.

Thanks.

[dtchepak]

@MilesLin Thanks for letting us know. 👍 Best of luck tracking down the problem!