Coverlet: Tests fail if assembly is strong named

[clairernovotny]

I'm trying to use Coverlet against an assembly that is strong named. When running those tests on the .NET 4.6.1 target, I get many errors similar to this:

Failed   Tests.MinMax.Min_AsyncSelector_Int32_4
Error Message:
 System.TypeInitializationException : The type initializer for 'Tests.AsyncEnumerableTests' threw an exception.
---- System.IO.FileLoadException : Could not load file or assembly 'System.Linq.Async, Version=4.0.0.0, Culture=neutral, PublicKeyToken=94bc3704cddfc263' or one of its dependencies. Strong name signature could not be verified.  The assembly may have been tampered with, or it was delay signed but not fully signed with the correct private key. (Exception from HRESULT: 0x80131045)
Stack Trace:
   at Tests.AsyncEnumerableTests..ctor()
   at Tests.MinMax..ctor()
----- Inner Stack Trace -----
   at Tests.AsyncEnumerableTests..cctor()

Is there a way to make this work?

[clairernovotny]

Looks like this is caused by xUnit's app domains. For dotnet test, it can be disabled with the following argument and it works: -- RunConfiguration.DisableAppDomain=true

[MarcoRossignoli]

@tonerdo maybe we should document?

[AArnott]

The workaround I've used is to drop this app.config file into the test directory, which prevents xunit shadow copying.

If coverlet fundamentally has to change the assembly being instrumented, I don't see an alternative. But I wonder why this is necessary, as I think other coverage tools don't require this. Do they hook into the CLR's profiler APIs to rewrite IL on-the-fly instead of rewriting the assembly?

[clairernovotny]

@AArnott that config file works for .NET Framework based tests, but I don't believe it is used for the .NET Core-based ones. The RunSettings/ command-line parameter I mention above does work though.

[AArnott]

but I don't believe it is used for the .NET Core-based ones

That doesn't matter. This failure doesn't occur in .NET Core anyway because .NET Core never validates signatures on strong name assemblies. So a .NET Framework-only workaround is all that's required.

[MarcoRossignoli]

If coverlet fundamentally has to change the assembly being instrumented, I don't see an alternative. But I wonder why this is necessary, as I think other coverage tools don't require this. Do they hook into the CLR's profiler APIs to rewrite IL on-the-fly instead of rewriting the assembly?

Uh...I'm curious to know how do they do without touch asm, if you've more detail or some docs share please.

[AArnott]

Here's one introductory link : http://blog.monstuff.com/archives/000058.html

[AArnott]

I'm hitting this again, and the workaround doesn't apply, because the test method itself explicitly creates an appdomain and uses shadow copying in order to test that the assembly behaves properly in those conditions.
The only thing I can think of is to exclude this test method from a code coverage run, and then run the test in isolation without the /p:collectcoverage=true switch.

[MarcoRossignoli]

Thank's for reporting this Andrew, strongname is applied on your assembly?Do you have the key?
We had an interesting conversation with @SteveGilham on it https://github.com/tonerdo/coverlet/issues/525#issuecomment-524635110 and new Cecil seem supports strong naming on write.
Another thinking(out loud) is to avoid to support/follow/fix netfx issues and focus on future of .net core and vstest integration.

[AArnott]

Yes, we fully strong name sign as part of the build, so if your instrumenting step could refresh the signature that would work.
.NET Framework is still a force to be reckoned with in my universe. It's a tad early to dismiss it at the moment. That said, I've applied the workaround, which isn't too bad, so if you don't want to deal with it, that's OK.

[MarcoRossignoli]

Thank's for your opinion, it's very appreciated really.
I'm reading/listening on media the trends rumors and I don't have a strong opinion today.
We have a pair of issue on netfx and I'm a "bit" alone at the moment, so my idea is to work more on "default/expected coverlet enviroment", it's the default .net core cross code coverage tool AFAIK and it's the build-in choice for dotnet test so I'm putting more effort on better vstest integration and support better msbuild features(msbuild suffer of very time spending investigation issue https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test).
But as maintainer I don't forget anybody and I try to track and spend my time to improve this tool on every aspect and better I can, so sorry for these defects 😞
You can read discussion here and add your opinion https://github.com/tonerdo/coverlet/issues/683 https://github.com/tonerdo/coverlet/pull/704