Coverlet: Upgrading coverlet.msbuild to 2.6.2 fails builds on travis that worked with earlier versions

[AdmiringWorm]

I was trying to upgrade the coverlet.msbuild on a few of my projects, only to find out that the new version fails with a type conversion exception.

The builds that fail when trying to generate the coverage report are all running on travis (both linux and osx, and both is failing) with .NET Core 2.1.603 installed.

Here is the exception/stack trace that gets outputted to the console

Exception/Stack trace

/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error : Could not convert string '{"Line":19,"Ordinal":0}' to dictionary key type 'Coverlet.Core.Instrumentation.BranchKey'. Create a TypeConverter to convert from the string to the key type object. Path 'Results[0].Documents['/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov/CodecovRunner.cs'].Branches.{"Line":19,"Ordinal":0}', line 1, position 14191. [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateDictionary(IDictionary dictionary, JsonReader reader, JsonDictionaryContract contract, JsonProperty containerProperty, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.CoveragePrepareResult.Deserialize(Stream serializedInstrumentState) in C:\Users\toni\Workspace\coverlet\src\coverlet.core\CoveragePrepareResult.cs:line 21 [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]
/home/travis/.nuget/packages/coverlet.msbuild/2.6.2/build/coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 83 [/home/travis/build/cake-contrib/Cake.Codecov/Source/Cake.Codecov.Tests/Cake.Codecov.Tests.csproj]

And the links I can share to the failing builds:
https://travis-ci.org/cake-contrib/Cake.Codecov/builds/542339897
https://travis-ci.org/cake-contrib/Cake.Transifex/builds/542340613
https://travis-ci.org/cake-contrib/Cake.Warp/builds/542341393

THe same projects are able to run coverlet.msbuild on appveyor, which leads me to believe there may have been a change in the latest version that would require to update the .NET Core SDK version (but I have not verified this yet).

[tonerdo]

@MarcoRossignoli can you pls help out? Thanks!

[MarcoRossignoli]

@AdmiringWorm can you try to upgrade Newtonsoft.Json to last version on tests project?
With version >= 10.0.1

[AdmiringWorm]

@MarcoRossignoli I added a reference to the latest version of Newtonsoft.Json (version 12.0.2), unfortunately this did not help and exactly the same error was reported.

[MarcoRossignoli]

Please try to "update" also all newtonsoft reference for all solution.

Can you also attach result of dotnet --info

[MarcoRossignoli]

Is the project OSS so I can in case repro locally?

[AdmiringWorm]

@MarcoRossignoli I didn't have any existing references for Newtonsoft.Json, so there is no additional references to update.

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.603
 Commit:    ae71c68742
Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.603/
Host (useful for support):
  Version: 2.1.11
  Commit:  d6a5616240
.NET Core SDKs installed:
  2.1.603 [/usr/share/dotnet/sdk]
.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.11 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[AdmiringWorm]

yes, it is open source: https://github.com/cake-contrib/Cake.Codecov

The updating of coverlet.msbuild is only in the following PR though: https://github.com/cake-contrib/Cake.Codecov/pull/43

[MarcoRossignoli]

Ok I'll try to repro locally

[MarcoRossignoli]

For now can you try to use dotnet tool or collectors, I'm curious to know if hit same issue...if you cannot don't worry!

[MarcoRossignoli]

@AdmiringWorm FYI https://github.com/tonerdo/coverlet#vstest-integration

[MarcoRossignoli]

@AdmiringWorm I get same issue using your .net sdk version...can you try with more recent?Or setting travis to use more recent SDK(2.2.x)?

F:\git\Cake.Codecov\Source\Cake.Codecov.Tests (develop -> origin)
λ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.2.100
 Commit:    b9f2fa0ca8
...
F:\git\Cake.Codecov\Source\Cake.Codecov.Tests (develop -> origin)
λ dotnet test /p:CollectCoverage=true /p:Exclude="[xunit*]*" /p:Configuration=Release
Build started, please wait...
Build completed.

Test run for F:\git\Cake.Codecov\Source\Cake.Codecov.Tests\bin\Release\netcoreapp2.0\Cake.Codecov.Tests.dll(.NETCoreApp,Version=v2.0)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.58]     Cake.Codecov.Tests.CodecovRunnerTests.Should_Use_Codecov_Runner_From_Tool_Path_If_Provided_On_Unix [SKIP]
Skipped  Cake.Codecov.Tests.CodecovRunnerTests.Should_Use_Codecov_Runner_From_Tool_Path_If_Provided_On_Unix

Total tests: 33. Passed: 32. Failed: 0. Skipped: 1.
Test Run Successful.
Test execution time: 1,3460 Seconds

Calculating coverage result...
  Generating report 'F:\git\Cake.Codecov\Source\Cake.Codecov.Tests\coverage.json'

+--------------+------+--------+--------+
| Module       | Line | Branch | Method |
+--------------+------+--------+--------+
| Cake.Codecov | 100% | 100%   | 100%   |
+--------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 100% | 100%   | 100%   |
+---------+------+--------+--------+

Test run for F:\git\Cake.Codecov\Source\Cake.Codecov.Tests\bin\Release\net46\Cake.Codecov.Tests.dll(.NETFramework,Version=v4.6)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
[xUnit.net 00:00:00.61]     Cake.Codecov.Tests.CodecovRunnerTests.Should_Use_Codecov_Runner_From_Tool_Path_If_Provided_On_Unix [SKIP]
Skipped  Cake.Codecov.Tests.CodecovRunnerTests.Should_Use_Codecov_Runner_From_Tool_Path_If_Provided_On_Unix

Total tests: 33. Passed: 32. Failed: 0. Skipped: 1.
Test Run Successful.
Test execution time: 1,4627 Seconds

Calculating coverage result...
  Generating report 'F:\git\Cake.Codecov\Source\Cake.Codecov.Tests\coverage.json'

+--------------+------+--------+--------+
| Module       | Line | Branch | Method |
+--------------+------+--------+--------+
| Cake.Codecov | 100% | 100%   | 100%   |
+--------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 100% | 100%   | 100%   |
+---------+------+--------+--------+
| Average | 100% | 100%   | 100%   |
+---------+------+--------+--------+


F:\git\Cake.Codecov\Source\Cake.Codecov.Tests (develop -> origin)
λ

[AdmiringWorm]

I noticed you opened a pull request for bumping the .net core version, I was just about to do that now (was working on something else, so couldn't do it until now).

EDIT:
I do believe that bumping the version will work (as mentioned in the initial issue), but I can not do this as it would change the minimum version I recommend/support for the project (as that .net core sdk version is no longer tested anymore).

[AdmiringWorm]

@MarcoRossignoli updating the sdk to 2.2.300 did indeed fix the issue on travis.

[MarcoRossignoli]

@AdmiringWorm can you use new coverlet dotnet tool until we fix?
Seem that some .net skd load old version Newtosoft.Json that doesn't support converters in a right way.
EDIT: should be supported by 2.1 ver https://docs.microsoft.com/en-us/dotnet/core/tools/global-tools

[AdmiringWorm]

@MarcoRossignoli I could do that of course, but I believe I rather will skip the 2.6.2 version, and wait for the next one instead.

I would need to implement my own custom logic if I change to the dotnet tool, which I don't really have the time to do right now (currently using Cake.Coverlet during build, but it only supports debug builds when running the dotnet tool).

[MarcoRossignoli]

Fair, sorry for the inconvenience :( I'll work on a fix asap.

[AdmiringWorm]

Fair, sorry for the inconvenience :( I'll work on a fix asap.

No worries, I would probably never have noticed this if it weren't for dependabot (I am lousy with updating dependencies manually, so it is good to rely on bots).

[JSkimming]

I've had exactly the same issue, happy to help with testing when #458 is ready.

Here's the affected PRs

• JSkimming/tesla-net#51
• JSkimming/abioc#87
• JSkimming/Castle.Core.AsyncInterceptor#68
• AutoTestNET/AutoTest.ArgumentNullException#43

[MarcoRossignoli]

@JSkimming as soon as @tonerdo do review/merge you'll can test with nightly build, thank's for help!

[vncoelho]

The same happens in NEO repo:

https://github.com/neo-project/neo/blob/6de8997a020088fe44c933a93ffb389c2605cdd9/.travis.yml

script:
 - dotnet restore
 - find * -name *.csproj | xargs -I % dotnet add % package coverlet.msbuild
 - dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

[MarcoRossignoli]

for now if "possible" one solution is to pin sdk to use on CI to 2.2.300 using maybe a global.json

[vncoelho]

I am quite bad in these Travis integrations, @MarcoRossignoli.
However, thanks for the recommendation and your attention.

If you have any example for me to update the script. ahueahuea

[MarcoRossignoli]

@vncoelho if you can change https://github.com/neo-project/neo/blob/master/.travis.yml#L11 to 2.2.300

[vncoelho]

Working, @MarcoRossignoli, thanks! "Your first commit" on Neo Project...aehuahuea

[MarcoRossignoli]

Working..

Glad to hear!

[MarcoRossignoli]

@dannyBies have tried upgrading sdk build version(if you can)?

[dannyBies]

Unfortunately I can't and do not have the time to properly investigate more. For now I'll downgrade to an earlier version, thanks for the help!

[SteveGilham]

Another data-point displaying the same symptoms : upgrading dependencies starting from https://github.com/SteveGilham/altcover/commit/a7f42444e3f46ac1f1fd4f7104b4e00bb6e5b17a

This builds on SDK 2.1.503 as the highest version that's both documented as on AppVeyor and comes with just naturally upgrading Visual Studio.

With Cecil 0.10.3 and coverlet.msbuild 2.6.1 -> 2.6.2, all is well, but upgrading Cecil to 0.10.4 as well causes the same symptom as in the original report. Also Cecil 0.10.3 -> 0.10.4 and coverlet.msbuild 2.6.1 works as at https://github.com/SteveGilham/altcover/commit/a30807a55e15259e2853ba23903721766a086f21 -- travis build https://travis-ci.org/SteveGilham/altcover/builds/546897076

[JSkimming]

TL;DR

PR #458 has fixed the issue I was experiencing.

Summary

I've just taken the nightly build in this PR JSkimming/Castle.Core.AsyncInterceptor#72 and all the tests now pass where previously I had this issue.

Nice work @MarcoRossignoli 👍