Coverlet: System.BadImageFormatException when code coverage is enabled.

[sesispla]

I am trying to add code coverage to kubernetes-client-csharp.

Desperate of not having official support from Microsoft I am giving a try to coverlet. I've created a feature branch from project's master and added Coverlet (everything fine in Travis so far.

But, as soon as I enable the "/p:CollectCoverage=true", I start receiving the following error in 80% of tests we execute:

[xUnit.net 00:00:01.7251872]       System.BadImageFormatException : [/home/travis/build/sesispla/kubernetes-client-csharp/tests/bin/Debug/netcoreapp2.0/KubernetesClient.dll] No string associated with token.

Full log from faulty TravisCI job

We're currently invoking the code coverage with this bash script:

#!/usr/bin/env bash

# Exit on any error
set -e

# Ensure no compile errors in all projects
dotnet restore
dotnet build --no-restore

# Execute Unit tests
cd tests
dotnet test --no-restore --no-build /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:CoverletOutput=./lcov.info
if [[ $? != 0 ]]; then
    exit 1
fi

Also happens locally with MacOS 10.12.6 High Sierra.

If I set /p:CollectCoverage=false tests passes properly in both TravisCI and locally.

Any clue on what's going on?

Thanks you!

[tonerdo]

Hi @sesispla, kindly point me to the feature branch and instructions on how to build the project and I'll take a look. Off the top of my head though, I'll say the problem comes from Mono.Cecil generating a bad assembly, I'm still using a beta version of Mono.Cecil so maybe updating it will fix this

[sesispla]

Here is the branch where I am currently working at:

https://github.com/sesispla/kubernetes-client-csharp/tree/feature/codecov

To build and execute the tests you can execute ci.sh or:

cd tests
dotnet restore
dotnet build
dotnet test

Thx!

[tonerdo]

A Mono.Cecil version upgrade didn't fix the issue like I hoped it will. Will have to take a closer look at figuring out that specific error message. From a quick skim though, it looks like it might have something to do with architecture. Although, I can't really be sure at this time.

[tonerdo]

@sesispla quick question, is your assembly signed?

[sesispla]

AFAIK, not yet. Assembly sign is a being discussed and pending due to non-windows lack of support.

[tonerdo]

This error also gets thrown on Windows

[sesispla]

Merged latest version from kubernetes-client-csharp and installed coverlet 1.0.2. The problem persists and is even worse. Some additional info:

1. We are using dotnet SDK 2.1.300.preview1-008174
2. Error is consistent through platforms (Windows, MacOS and Linux)
3. Now Travis-CI gets stuck and do not finish.
4. Local coverage test takes 20 minutes.
5. Test results from local tests:
   dotnet 2.0 test results: Total tests: 63. Passed: 7. Failed: 56. Skipped: 0.
   dotnet 2.1 test results: Total tests: 61. Passed: 7. Failed: 54. Skipped: 0.
6. Error is the same: No string associated with token.

Build: https://travis-ci.org/sesispla/kubernetes-client-csharp/builds/363457392

[brewsoftware]

Hey, we are seeing a similar behaviour with one of our projects. I am trying to diagnose to give some more information but I'm not sure where the issue is at the moment.

We are running in a microsoft/dotnet:2.0.0-sdk docker container.

[tonerdo]

Hmm...this is definitely taking me for a loop. I still have a local copy of the kubernetes-client-csharp and I do get the same error. There have been a couple of changes to improve performance that I'm yet to release but I haven't been able to figure out the No string associated with token error

[yamac-kurtulus]

Hey, We recently solved a similar issue. The build should be in full debug mode. Make sure you are not running the test with -c release option. Your default configuration might be set to release.

[brewsoftware]

So I've had a bit of a look into this and I think the problem is in Cecil or at least the way the read/write is loading and saving the Dll format metadata (x86 vs x64).

https://github.com/jbevain/cecil/blob/33cfa97b69dcce2cfed2b15ef0633cd8f8c23bd1/Mono.Cecil/AssemblyWriter.cs#L84

I have noticed that the assembly that can't be loaded is being changed from a PE x86 (Mixed mode) assembly to something else (through a hex diff).

Also, if I change the output of the test project to x64 then xunit fails to load the assembly..

Still stumped but I'm leaning towards a x64 vs x86 issue still.

[tonerdo]

Ah thanks for the observation @brewsoftware. Probably doesn't make it easier to fix, but at least it points me in the right direction

[queen-of-code]

I'm experiencing this as well, running on Ubuntu with dotnet sdk version 2.1.302 and using the latest version of the coverlet global .net tool. Our builds/tests are specifically done as --configuration Debug.

We've got something like 30 test assemblies, and it's only happening with a few. I haven't dug too deeply into them as of yet to see what the commonalities might be.

[queen-of-code]

@tonerdo is there anything I can do to help debug? I'd love to get this resolved, as it's the last blocker I have to switching our build machines over to linux. I've tried upgrading all the relevant nuget packages. It's just one our of test assemblies that's doing it, but of course it's the biggest.

[tonerdo]

@sesispla @queen-of-code sorry for taking so long to get back to this. I've done some digging and found what seems to be a string encoding issue. Some of the string constants that are injected during the instrumentation process for some reason aren't encoded accurately. While I'm still unable to find out _why_ this happens or what kind of assemblies it happens to, I'm working up an alternate approach to register line hits that totally bypasses this issue.

I should be able to come up with and push out a stable fix by tomorrow once I'm able to achieve a successful run with my local version of @sesispla's kubernetes-client-csharp project.

There's a chance I might get sidelined with other tasks, so I'll appreciate if you hold me accountable on my promise to get this fixed ASAP

[pjanotti]

Unrelated to this problem but related to:

I'm working up an alternate approach to register line hits that totally bypasses this issue.

I was preparing a simple and naive perf improvement using a string comparer that leveraged the likely location of diffs (the last few chars) in the strings passed to the "marker". However, it is clear that was possible to do even better by changing how the marker receives hits. In light of the comment above my initial change doesn't make sense, I will wait on the new approach to see how the perf is affected for the case that I profiled (System.Text.Encodings.Web.Tests in corefx).

[tonerdo]

@sesispla @queen-of-code this issue has been fixed in the latest release: https://www.nuget.org/packages/coverlet.msbuild/2.2.1, also includes your Include changes @pjanotti. Let me know if you run into any other issues

[pjanotti]

@tonerdo I did some perf measurements for this change against coverage runs for System.Text.Encodings.Web.Tests in corefx in got about 43% of reduction in running time for this project, pretty good 👍 !

I have some low hanging fruit changes that get a bit more on top of that, I will create a PR with those no later than tomorrow.

[queen-of-code]

Can confirm that this is working now (though I'm using the dotnet tool install version).

[tonerdo]

@queen-of-code that's great to hear. It should work regardless of the coverlet mode (msbuild or global tool) you use

[queen-of-code]

@tonerdo Now that that's cleared out the bulk, I'm able to see a few other failures. Is netcoreapp 2.0 a requirement? I didn't see it as such on the nuget package, but perhaps I missed something. I have a number of netcoreapp1.0 tests failing with

System.IO.FileNotFoundException : Could not load file or assembly 'netstandard, Version=2.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

   at API.Common.set_ApplicationId(Int32 value)
   at API.Test.FilterTests() in /opt/teamcity/work/28c9a0afce38408c/test/unit/Api.Test/UserActivityApiTests.cs:line 581

They only fail when I do code coverage runs via the tool, to be clear. When run the usual way via dotnet test, they pass fine.

[tonerdo]

@queen-of-code I believe all .NET Core global tools require netcoreapp2.0 and above as they are deployed as framework dependent apps

[sesispla]

Coverage up and running :)

https://coveralls.io/github/sesispla/kubernetes-client-csharp

Now I have an issue with relative paths but it looks like a Coveralls.Net issue.

Thx @tonerdo!

[tonerdo]

That's great to hear @sesispla! For the relative path issue, I ran into something similar in Coveralls. You need to go into your settings and set the repo root folder

[tonerdo]

I'm closing this out now. Thanks for your patience on this everyone