Nunit-console: Exception encountered unloading AppDomain

[Trass3r]

We tested the fix for #168 via the 3.6.1 NuGet release but still get other sporadic failures:

nunit3-console.exe ..... --work=.../Release --out=TestOutput.log --result=TestResult.xml --labels=On --framework=net-4.5 --dispose-runners --agents=8

Unhandled Exception: NUnit.Engine.NUnitEngineException: Exception encountered unloading AppDomain
Agent Process was terminated successfully after error.
    at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
    at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
    at NUnit.Engine.Runners.TestExecutionTask.Execute()
    at NUnit.Engine.Runners.ParallelTaskWorkerPool.ProcessTasksProc()
    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart()

[ChrisMaddock]

I'm not sure this is related to the failures fixed in #168. Is this new for you since 3.6?

More information should be written to the log in this case. Can you try to reproduce with --trace=Error - and post the relevant bit of log?

@nunit/engine-team
Looking at: https://github.com/nunit/nunit-console/blob/651de525ffe1f448d3d6c31ea645846338397baf/src/NUnitEngine/nunit.engine/Services/DomainManager.cs#L177
From what I can find out CannotUnloadAppDomainException will be thrown if the user code can not be terminated immediately, e.g. if finalizers are running. Should we perhaps be retrying the unload over 30 seconds or so? There's a 30 second timeout, but that's only for a hang - whereas this would throw immediately and crash, as far as I can see?

[Trass3r]

Yes 3.5 works fine. 3.6.0 had issue #168. With 3.6.1 it's this.

[Trass3r]

The log doesn't contain much: 12:43:57.140 Error [80] DomainManager: Unable to unload AppDomain domain-f69c0556-FooUnitTest.dll

[julienadam]

I started getting the same error on our build chain recently. The commit that broke the build was a change to the nunit console parameters to set workers and agents to 1 to try and debug a concurrency issue in some of our tests. We've been running 3.6.1 successfully previously.

I've bumped the tracing level to Verbose but don't get anything significant.

14:27:37.338 Info  [ 1] ProcessRunner: Done running xyz.dll
14:27:37.338 Info  [ 1] ProcessRunner: Unloading xyz.dll
14:27:48.395 Warning [ 1] ProcessRunner: Failed to unload the remote runner. Exception encountered unloading AppDomain

As a side note it's a bit of a shame to only log exception messages in the internal logs. Here we don't get any information at all since apparently the exception has no message. Logging exception.ToString() would be a lot more useful IMHO.

[ChrisMaddock]

There's a separate issue to improve the logging here at: https://github.com/nunit/nunit-console/issues/111

Doing that issue would probably be a help in working out what's going on here.

[mletterle]

I just ran into this with our integration test suite and I'm fairly certain it's because of the 30 second time out, would be handy to have a configurable override for that setting at least...

[julienadam]

I'm traveling at the moment but here's a quick update. I debugged the
process on the CI machine and found the culprit in our tests. One of them
was holding an active thread that was blocking the unloading.

I think a good option would be to dump the callstacks of all threads of the
agent processes before throwing the unload exception. That would help a lot
to find what is going on.

Cheers

Le 12 avr. 2017 1:51 AM, "Michael Letterle" [e-mail] a
écrit :

I just ran into this with our integration test suite and I'm fairly
certain it's because of the 30 second time out, would be handy to have a
configurable override for that setting at least...

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nunit/nunit-console/issues/191#issuecomment-293432900,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABrccs7xVM02g5kFDOaHOwusk1ckZ1d7ks5rvBIfgaJpZM4MZOOh
.

[jaytonic]

I also got this issue, but I'm not sure why, only information I've is this:

01:56:16.397 Exception encountered unloading AppDomain
01:56:16.397 Agent Process was terminated successfully after error.
01:56:31.573 Committing...

Did you found any workaround?

[julienadam]

I did not find a workaround but I found the reason the unloading was failing. And it was in our code.

To find out what was blocking I logged on remotely to our build server, ran Visual Studio on it and debugged the NUnit processes directly on the machine.

[julienadam]

Follow up on this. We've had another similar issue that cropped up on our (rather big) build chain. This time the issue is in third party code we can't really fix. So I ended up forking NUnit to remove the unload exception altogether, as an experiment. Builds have been consistently successful since then.

It might make sense to downgrade from a fatal exception to something milder? I definitely think it should bubble up so that users are aware of it but throwing means that no test report is generated at all.

Maybe it should still write a report but mark it as failed with the CannotUnloadAppDomainException error message in the report? This way the tests results would be available but the error would be plainly visible.

[CharliePoole]

The trick here is that the tests have already been run and the result reported before this error actually occurs. Since the problem happens "between" test runs - in the case of the console after the sole test run - there's no place to report it except in an exception of some kind. So, it comes down to a matter of how the console handles the exception.

NUnitEngineException is used to report errors in the engine, which usually mean the test cannot be run. Perhaps we should not be wrapping this exception but let the console handle the CannotUnloadAppDomainException and just issue a warning.

[jnm2]

but let the console handle the CannotUnloadAppDomainException and just issue a warning.

NCrunch does this IIRC

[mafshin]

I have the same issue, after passing all tests, I get

Warning: System.Threading.ThreadAbortException: Thread was being aborted.
   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
   at NUnit.Engine.Services.DomainManager.DomainUnloader.UnloadOnThread()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Warning: Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests

Warning: Unable to unload AppDomain, Unload thread timed out.

Nunit Version: 3.7.0

[Trass3r]

Just fyi, we worked around this problem by fixing the root cause of the "unable to unload" problem.
In our case an unexpected exception was thrown and caused our code to hang because it was not exception safe (no finally block used for signaling event).

[ChrisMaddock]

Thanks for the update @Trass3r - good to know your up and running.

For this issue - I think @CharliePoole's point is worth investing some time into, this could suitably be handled as a warning by the console, and still allow the results to be written.

[Fernell]

Same error here on nunit3-console v3.7. I've removed throw new NUnitEngineException("Exception encountered unloading AppDomain", _unloadException); in DomainManager.cs and it works like a charm. It seems it is enough to log some warning instead of throwing this exception.

PS. In my case downgrading to v3.5 did not work.

[CharliePoole]

@Fernell Funny thing, you've put us back to our traditional approach of ignoring these errors. We added an exception when people complained about the silent failure. We've been tinkering with it ever since.

[Fernell]

Well, you can not satisfy everyone. I was struggling with this error for some time because our code seems to work properly (all tests are passing when I commented this exception) and I was unable to find out why it crashes anyway. Perhaps it should be configurable if exception should be thrown or not (like some --silent-domain-unload-failure flag). Anyway sorry for being troublemaker here.

[CharliePoole]

No bother... I preferred the old approach myself. However, it causes a problem when a large number of appdomains can't be unloaded and continue to take up memory. They may also be holding other resources that are needed by the tests.

Personally, I would favor removing the exception in order to design some better way to notify the user of the problem.

[jaytonic]

100% agree too. IMHO, it's better to maybe loose a notification than just prevent the whole tests to execute.

[ChrisMaddock]

NUnitEngineException is used to report errors in the engine, which usually mean the test cannot be run. Perhaps we should not be wrapping this exception but let the console handle the CannotUnloadAppDomainException and just issue a warning.

This suggestion of yours is still the best idea imo @CharliePoole. Just needs someone to do a pull request for it... @Fernell or @jgrossrieder, either of you interested? 😄

[Rajkumar-Uppala]

Hi Team,
We are trying to Upgrade our Projects from Nunit 2.6.3 to 3.6.1

We are facing an issue with "Exception encountered unloading AppDomain" then it is terminating with "Agent Process was terminated successfully after error" .
We are using Teamcity for automated test run.
Few blogs here says that the issue might be due to 'long running test cases' or 'any unhandle exceptions during test running'.
Not sure how to identify them as we are not facing the issues on local Dev machines.

Could anyone of please advise me how to overcome this.
1.Is their any workaround to solve this?
2.Is their any suggestion -that this issue could be solved by making any changes to project?
3.Is their any commands that can be used while running test cases through TeamCity which will suppress/solves the issue without failing the build.
4.Can we know in which version of Nunit this issue will get solved.

Any advise on this would be greatly helpful for to proceed with upgradation of Nunit to our Projects.

[14:28:13][Step 8/9] Errors, Failures and Warnings
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] 1) Error : C:\BuildAgent\temp\buildTmp\2fd5jXcXtHuxVQRYgOilFKdUX6jHeD53.nunit
[14:28:13][Step 8/9] Exception encountered unloading AppDomain
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] Server stack trace:
[14:28:13][Step 8/9] at NUnit.Engine.Services.DomainManager.DomainUnloader.Unload()
[14:28:13][Step 8/9] at NUnit.Engine.Services.DomainManager.Unload(AppDomain domain)
[14:28:13][Step 8/9] at NUnit.Engine.Runners.TestDomainRunner.UnloadPackage()
[14:28:13][Step 8/9] at NUnit.Engine.Runners.AbstractTestRunner.Unload()
[14:28:13][Step 8/9] at NUnit.Engine.Runners.AbstractTestRunner.Dispose(Boolean disposing)
[14:28:13][Step 8/9] at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
[14:28:13][Step 8/9] at NUnit.Engine.Runners.AggregatingTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
[14:28:13][Step 8/9] at NUnit.Engine.Runners.AbstractTestRunner.Run(ITestEventListener listener, TestFilter filter)
[14:28:13][Step 8/9] at NUnit.Engine.Agents.RemoteTestAgent.Run(ITestEventListener listener, TestFilter filter)
[14:28:13][Step 8/9] at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs)
[14:28:13][Step 8/9] at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Object[]& outArgs)
[14:28:13][Step 8/9] at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg)
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[14:28:13][Step 8/9] at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
[14:28:13][Step 8/9] at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] Test Run Summary
[14:28:13][Step 8/9] Overall result: Failed
[14:28:13][Step 8/9] Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
[14:28:13][Step 8/9] Start time: 2017-10-13 13:25:11Z
[14:28:13][Step 8/9] End time: 2017-10-13 13:28:00Z
[14:28:13][Step 8/9] Duration: 169.729 seconds
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] Results (nunit3) saved as C:\BuildAgent\temp\buildTmp\2fd5jXcXtHuxVQRYgOilFKdUX6jHeD53.nunit.xml
[14:28:13][Step 8/9]
[14:28:13][Step 8/9] Exception encountered unloading AppDomain
[14:28:13][Step 8/9] Agent Process was terminated successfully after error.
[14:28:14][Step 8/9] [JetBrains dotCover] Analyzed application exited with code '-100'
[14:28:17][Step 8/9] Process exited with code -100

Thanks & Regards,
Rajkumar Uppala.

[Rajkumar-Uppala]

@ChrisMaddock - thank you for your advise on this
You have stated that
How are you running the tests locally, where you can't reproduce? I suggest you try fetching the .nunit project file generated by TeamCity, and see if you can reproduce the issue using that, and try on the same machine as TeamCity is running.

Could you please help me to elaborate how to do this.
I am bit confused on your advise - how to refer/use the project file that teamcity generates and how to make use of it to replicate the issue on my local machine using Visual Studio?

Please advise

Thanks & Regards,
Rajkumar Uppala.

[ChrisMaddock]

@Rajkumar-Uppala Sorry, I'm not a TeamCity user myself, but see 'debugging nunit tests' on this page.

https://confluence.jetbrains.com/display/TCD10/Getting+Started+with+NUnit#GettingStartedwithNUnit-DebuggingNUnittests

[ChrisMaddock]

You'll want to pass that .nunit file into the NUnit Console - as TeamCity is doing. Visual Studio shouldn't come into it.

If you're currently running in the visual studio test window, you won't see this exception due to the way tests are run. Hopefully running in the console may give you a repro that you can they debug.

[Rajkumar-Uppala]

@ChrisMaddock - Thank you for the advise.
I am not familiar with Nunit Console. Will try to explore this and see how it helps.
Meanwhile if you get any clue please do share with us.

[Rajkumar-Uppala]

Hi Everyone,
After lot of efforts, we found that the issue is with our code which is troubling.
With Nunit 2.6.1 it used to consider it as warning. But with Nunit 3.6.1 it is considering as Error.
Rootcause of the issue in our logic is due to one of the Nunit test cases is referring to actual async await service method instead of mock services.
As the test case is waiting for the result and will never get the response. Because of this at the time of test completion and still it is waiting for response .... it is throwing us this error.

With the implementation of mock service instead of referring to actual service solves the issue.

Hope my investigation will help someone to investigate their issues considering this corner also.

Thanks & Regards,
Rajkumar Uppala.

[ChrisMaddock]

@Rajkumar-Uppala - thanks for following up. You're correct this has changed from a warning to an error - as you'll see above, we're actually considering changing it back!

In my opinion, it's important to flag it to the user in a non-ignorable way, so that the user can make some attempt to fix the issue. As you've seen - it often highlights a more fundamental issue in user-code. It would be better if we could write the results file out however, and just exit with a non-zero code.

[Rajkumar-Uppala]

@ChrisMaddock - thanks for your comments on my reply.
I would like to add a point here for the benefit of all.

Please try to implement a process to include more information in detail as part of release notes/breaking changes (If there is any possibility) by which everyone will be benefited.

• Please try to prepare some remediation steps to resolve issues (if you know what might create issue as part of upgrade)
  For example : Let us consider current discussion. Many of them are facing this issue due to different reasons. if you can consolidate the solved way of addressing the issue either part of breaking changes/release notes or providing some links

Hope you understand my concern behind raising this.

[ChrisMaddock]

It's a bit difficult to suggest remediation for a problem in user code! 🙂 As you say, the cause of your fail-to-unload is likely different to everyone else's cause in this thread!

We recognise it's currently not ideal though - so what we're doing to improve this experience:

• https://github.com/nunit/nunit-console/issues/191 - Continue writing test results when app domain fails to unload
• https://github.com/nunit/nunit-console/issues/111 - Providing more debug info when app domain fails to unload

NUnit is open source software however - all contributions welcome! So if you have any concrete ideas on how we can improve things, and the team agrees with your suggestion, please share them and we can improve things for everyone! 😄

[Rajkumar-Uppala]

@ChrisMaddock - I accept and can understand difficulty or pain to implement my suggestion.
My intension behind raising this is for the benefit of all and sharing my thoughts as I felt that this is the best platform where I got some useful suggestions which helps me to investigate the issue in different approach. May be you might get a chance to implement better way of documentation by considering all our suggestions/concerns/observations and etc...

Thank you everyone in the thread for sharing your best thoughts against the issues raised here .. for me @ChrisMaddock gives his best.... to address this thread issues

[ChrisMaddock]

Been taking a look at this today. A key thing I don't think we've been aware of so far - results only seem to be lost when using an .nunit project file. I imagine that's related to https://github.com/nunit/nunit-console/issues/116

I'll loop round to look at that problem later.

[jnm2]

@ChrisMaddock wow, #116 is old! I was so unfamiliar with everything back then. Would you like to take it since I have other stuff going on? 😁 😁

[ChrisMaddock]

Please! Was just having more of a dig - I'm not even totally sure they are the same bug right now, I think it may just be that #116 causes a specific case to be exposed which would be a bit more niche otherwise. 😄

[ChrisMaddock]

Ah, it is.

This can be repro'd with:

nunit3-console "Issue191.dll" "Issue191.dll" --agents=1

116 is only involved, because that's effectively causing the NUnit Project loader to do the above. Two separate bugs really. 😄

[jnm2]

116 is dead last on my NUnit todo list, just so you know. 😁

[ChrisMaddock]

No worries - I'll see how I get on!

116 I'm less worried about, nobodies really been chasing that one. I wanted to get a fix in for this issue, as it keeps attracting people at the moment. 😄

[ghost]

Hi Guys. I'm still having some issues related to this thread in version 3.8 of Nunit Console.

We just upgraded from version 3.5 to 3.8. The issue is that it looks like it can't find the dll "websocket-sharp" but that does not change the result of our tests

I saw the solution #321 and some other related threads, but could not find an option to turn this exception back to an warning. Quoting @CharliePoole " I would favor removing the exception in order to design some better way to notify the user of the problem.", could you guys please tell me if that was implemented? Is there any way to change this exception back to a warning?

At the moment, I'm trying to fix the websocket-sharp error, but if there is an option to turn the exception back to a warning it would be much better, since this kind of errors are not an issue to us

[ChrisMaddock]

@hbsis-williamdasilva - See https://github.com/nunit/nunit-console/issues/403. Current solution is to use the build of master from MyGet (see details in README) - this will revert to existing zero in the next release.

[ghost]

@ChrisMaddock Thanks for the reply.
I downloded the latest version from master at MyGet (https://www.myget.org/feed/nunit/package/nuget/NUnit.ConsoleRunner/3.9.0-dev-03972)

And got the nunit3-console.exe

I'm using this console to run the dll that I got the error with v 3.8, as we can see here

But I'm still getting the error

Am I missing something?

[ChrisMaddock]

@hbsis-williamdasilva Ah sorry, I misread - I believe you're seeing a separate exception thrown when trying to print app domain details. Would you file a new issue for this?

Apologies!

[ghost]

Sure, thanks for the help. I'll create a new issue

[ghost]

@ChrisMaddock Issue created #410