Vstest: Test discovery leaves too many files locked - causes problems for metaprogramming

[mgravell]

Description

When using dotnet test (and presumably other approaches), the discovery phase locks dll files, which are kept locked for the duration of the execution phase. This means that if any of the files are actually outputs (meta-programming artefacts), they will prevent the second (etc) test runs from executing successfully

Steps to reproduce

• write a test that emits a dll (meta-programming, AssemblyBuilder etc) to the test folder during execution
• run the test (file is generated)
• run the test again (file cannot be regenerated as it is locked by discovery)

A more complete set of repro steps (including command-line and git commit) is in this report (I initially filed against xunit, and was redirected here)

Expected behavior

Subsequent test runs succeed, via one of (I have no strong views on which):

• not locking files between discovery and execution
• allowing an inclusion/exclusion filter from dotnet CLI to prevent unwanted locks from happening
• implementing shadow-copy so that it doesn't matter what is locked (in the transient folder)
• having obvious options (presumably in dotnet tooling hooks) to allow the test directory to be cleaned between runs

Actual behavior

File is locked; subsequent test runs fail

Environment

Windows 10 x64, dotnet 1.0.0, vstest.console 15.0.26228.0

[aelij]

One more option is to use System.Reflection.Metadata for discovery instead of Assembly.Load, then load only the assemblies that actually have extensions. Here's some sample code I've used for similar purposes.

[codito]

@mgravell I could clean the output folder between test runs by adding following Target to test project. Hope this will help workaround the issue.

  <Target Name="CleanGeneratedTestFiles" BeforeTargets="PrepareForBuild" Condition="'$(TargetFramework)' != ''">
    <RemoveDir Directories="$(OutputPath)" />
  </Target>

If I've a test project test.dll and it generates a.dll during test execution. Now a.dll is generated during first run, for the second run we observe a.dll is getting loaded during dotnet test invocation. So when the test actually executes, it cannot generate/overwrite a.dll again. Since a.dll is generated at runtime, test.dll doesn't depend on a.dll, so it's weird why a.dll should get loaded during second test run (for discovery or execution). Is this understanding correct?

From a conceptual standpoint, the testhost process (which executes the test) and test platform has no knowledge of any test framework, it doesn't load the assemblies (it's a bug if it does). Knowledge of test framework and finding the tests in an assembly is with an adapter. Some adapters can possibly use metadata approach suggested by @aelij, a few frameworks allow code to be executed during discovery (e.g. xunit), they may not be able to use it.

[aelij]

I think the test host loads assemblies in order to discover extensions (see here for example). Couldn't that also be a problem?

[Mnuzz]

See https://github.com/Mnuzz/AutoHotkey.FSharp/tree/31d34922e1163d1a1fe98fff2e66976df2f9039f for a (possible) self-contained reproduction of this issue from VS Test Runner. The issue I mention is not the same exact scenario, but it could be a similar mechanism at fault. The (native) DLL files that are being locked are not generated dynamically, but they are referenced as embedded resources. The files in question are AutoHotkey.dll or v2_AutoHotkey.dll. During build, these DLL files are copied to the output directory, and then at runtime they are dynamically loaded.

In VS2017, navigating to Test -> Test Settings -> Uncheck "Keep Test Execution Engine Running" is a workaround for my VS test runner issue. It helps confirm my suspicion that the VS Test runner process has trouble unloading these types of files. I think that this setting is merely a workaround and not a solid, layman-friendly solution. It certainly took me some time to realize that this setting was there, and even then I can see how it would be not obvious to every developer that this would fix it.

If a solid solution is to be found, it would be good to know where the offending code/library is. Is this a flaw in VS testrunner, and a separate flaw in dotnet testrunner, each with its own, separate implementation of this loading/unloading behavior? Or are both runners referencing a dependency that attempts to handle this?

I've implemented my own variant of shadow copying at one point, to solve a similar problem when developing extended plugins for a computer game, but it's definitely not industry quality. In case this helps at all, as a starting point, here is the example code.

What this did was, before each project was built, a random suffix was appended to the base assembly name in the csproj file, and each local upstream dependency reference was also updated to use the same suffix.After build, the project file should have been reverted back to normal, but I think that may not have been implemented.

[llehn]

The dll locking is absolutely annoying. Every build = fail. Need to kill discoveryengine first before build.

[mayankbansal018]

@mgravell The issue is because the TestMethods are creating some dll on runtime, & are unable to clean them up after they are executed, since the newly created dll's are loaded in the host process. This causes an issue during second run as the dll is already created, hence the Testmethod cannot create it again.
Can you please try the solution codito proposed.

@llehn the discoveryengine has been replaced by our new testplatform in VS2017.5

[mayankbansal018]

@mgravell , can you please review my above comment, & confirm if it works for you?

[mayankbansal018]

@mgravell , as per explanation above, I'm closing issue. Please reopen it if there is a need to discuss more on it.

[Aaronontheweb]

@mayankbansal018 I'm running the absolute latest version of Visual Studio 2017 and still run into this issue frequently with all of the XUnit assemblies being locked on build.

[smadala]

@Aaronontheweb Can you please create new issue with detail repro steps?