Runtime: No successful code coverage run in CI since July

[stephentoub]

The last successful run was on July 17th, https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/, over three months ago.

cc: @danmosemsft

[danmosemsft]

The runs have been generating valid data, but one or more tests have failed, so the badge does not update. To see latest data, go to here

https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/

find the most recent number, then construct this URL eg

https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/712/Code_Coverage_Report/

Having said that the build today is not getting results – MSBuild crashed (probably some build task). A couple months ago I made a change to MSBuild to include the callstack when this happens, but apparently we don’t have that change yet. Most likely this is sporadic and will go away tomorrow.

Looking at the test failures, I see this one, which seems consistent in the last two runs - any thoughts?

      System.Threading.Tasks.Tests.ExecutionContextFlowTest.TaskDropsExecutionContextUponCompletion [FAIL]
        System.TimeoutException : Task timed out after 60000ms
        Stack Trace:
          D:\j\workspace\code_coverage---b14a30e5\src\Common\tests\System\Threading\Tasks\TaskTimeoutExtensions.cs(54,0): at System.Threading.Tasks.TaskTimeoutExtensions.TimeoutAfter[TResult](Task`1 task, Int32 millisecondsTimeout)
          D:\j\workspace\code_coverage---b14a30e5\src\System.Threading.Tasks\tests\Task\ExecutionContextFlowTest.cs(61,0): at System.Threading.Tasks.Tests.ExecutionContextFlowTest.TaskDropsExecutionContextUponCompletion()

and this one, which may be because profiling is not compatible with coverage instrumentation:

      System.Runtime.Tests.ProfileOptimizationTest.ProfileOptimization_CheckFileExists [FAIL]
        'ProfileOptimization_CheckFileExists_19_a3f7e757' does not exist
        Expected: True
        Actual:   False
        Stack Trace:
          D:\j\workspace\code_coverage---b14a30e5\src\System.Runtime.Extensions\tests\System\Runtime\ProfileOptimization.netcoreapp.cs(49,0): at System.Runtime.Tests.ProfileOptimizationTest.ProfileOptimization_CheckFileExists()

and others which are likely sporadic eg

      System.Net.Http.Functional.Tests.SocketsHttpHandler_SchSendAuxRecordHttpTest.HttpClient_ClientUsesAuxRecord_Ok [FAIL]
        System.Net.Http.HttpRequestException : An error occurred while sending the request.
        ---- System.IO.IOException : Unable to read data from the transport connection: An established connection was aborted by the software in your host machine..
        -------- System.Net.Sockets.SocketException : An established connection was aborted by the software in your host machine.
        Stack Trace:
          D:\j\workspace\code_coverage---b14a30e5\src\System.Net.Http\src\System\Net\Http\SocketsHttpHandler\HttpConnection.cs(681,0): at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)

Historically we have hit crashes during test runs (not exceptions like above - https://github.com/dotnet/corefx/issues/29661) which Paulo determined we could only fix by moving off opencover to coverlet, which is tracked by https://github.com/dotnet/buildtools/pull/2184. As far as I know that is unblocked. @ViktorHofer took over code coverage ownership, but is currently fully occupied with Arcade conversion. So I don't think that will be looked at for perhaps a month.

[danmosemsft]

In general code coverage work should be being tracked in https://github.com/dotnet/corefx/projects/9

[danmosemsft]

@ViktorHofer is there a flag or some way I can cause ProfileOptimization_CheckFileExists to not run in code coverage runs?
cc @brianrob to confirm it's not surprising this fails when collecting code coverage.

[brianrob]

I'm not aware of any reason why ProfileOptimization_CheckFileExists should fail for code coverage runs, but I'll admit that I don't know very much about how code coverage runs work.

Adding @noahfalk who owns ProfileOptimization.

[MarcoRossignoli]

FYI we disabled test on some distros weeks ago https://github.com/dotnet/corefx/pull/32737 for similar issue

[danmosemsft]

Thanks @MarcoRossignoli but tht was specific to RedHat, whereas coverage is (currently - plans to add Linux) only on Windows.

[ViktorHofer]

@ViktorHofer is there a flag or some way I can cause ProfileOptimization_CheckFileExists to not run in code coverage runs?

The hacky solution I can think of would be to pass /p:xunitoptions to the code coverage run to disable that method.

[noahfalk]

@kouvel actually owns ProfileOptimization now

[danmosemsft]

@weshaggard it looks like this built with D:\j\workspace\code_coverage---b14a30e5\.dotnet\sdk\2.1.401\MSBuild.dll. When do we expect to update it, so it could include a change they merged Aug 20 ?

[weshaggard]

@weshaggard it looks like this built with D:\j\workspace\code_coverage---b14a30e5.dotnet\sdk\2.1.401\MSBuild.dll.

No we don't expect to update this SDK regularly. It will only be updated to a prerelease version if we absolutely need a fix for a release otherwise we will stay at the last released version of the toolset.

[danmosemsft]

Fair enough

[stephentoub]

The runs have been generating valid data, but one or more tests have failed, so the badge does not update.

Thanks, Dan. That's good to know. Can we get the badge to point to the latest run, successful or not, until this is addressed? Otherwise, it's very undiscoverable.

[stephentoub]

ExecutionContextFlowTest.TaskDropsExecutionContextUponCompletion

I'll see if I can repro locally.

[stephentoub]

I'll see if I can repro locally.

Having trouble reproing this locally, unfortunately, which suggests it's not just some strange interaction with coverage and this particular test.

[danmosemsft]

If there's a suspicion that TaskDropsExecutionContextUponCompletion is not actually stuck, but needs longer than 60 sec, we can use the approach of increasing the timeout, but making it fail and log the time taken if it's longer than 60 sec (actually this feature could be built into the remote executor code). I imagine it's unlikely it needs longer than 60 sec though... My other suggestion is to add some console writes so we know where it gets stuck.

[danmosemsft]

Can we get the badge to point to the latest run, successful or not, until this is addressed?

Probably not. @mmitche right now by some magic, https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/Code_Coverage_Report goes to the result of the last successful build. Is there a way to point instead to the result of the last build that completed? Eg., https://ci.dot.net/job/dotnet_corefx/job/master/job/code_coverage_windows/712/Code_Coverage_Report/ where 712 is something that means "whatever the last completed one was"

[ViktorHofer]

Closing with joy as fixed with https://github.com/dotnet/corefx/pull/33423