Nunit-console: NUnitEngineException : Unable to acquire remote process agent

[ChrisMaddock]

@niwe commented on Fri Apr 27 2018

I'm trying to start some tests with nunit3-console.exe (v3.8.0) on a virtual machine running Windows 10 (x64) but before any tests get executed NUnit stops. I get the following error messages:

Unhandled Exception: NUnit.Engine.NUnitEngineException: Failed to start remote test agent.
   at NUnit.Engine.Services.TestAgency.OnAgentExit(Object sender, EventArgs e)
   at System.Diagnostics.Process.OnExited()
   at System.Diagnostics.Process.RaiseOnExited()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)

And after a few more seconds this is shown:

1) Error :
NUnit.Engine.NUnitEngineException : Unable to acquire remote process agent
--NUnitEngineException
   at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunner()
   at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)

I recently added the flag --_x86_ to NUnit and that's when this issue started. I added the flag because I also added an external file (_Microsoft Exression Encoder_) to my test project, and it seems to be x86 only.
I read (on _Stackoverflow_ I think) that using --_inprocess_ works. I used to have that flag but it is not compatible with --_x86_. So I can't get it to work no matter how I try.

This is the command I use to run NUnit: nunit3-console.exe C:\...\GUITests.dll --x86 --where "cat == .NIGHTLY" --result="C:\...\Logs\TestResult.xml;format=nunit2"

---

@rprouse commented on Fri Apr 27 2018

Are any of the other assemblies that your tests reference or that your code references x64?

How did you install NUnit Console and what version are you using?

I will likely move this issue to the NUnit.Console repository, but would like to confirm here first.

---

@niwe commented on Mon Apr 30 2018

I'm not entirely sure how I can see if assemblies are x64 or not.

In my Visual Studio solution I have three projects. The project where NUnit is used (and Microsoft Expression Encoder) I only reference some standard .NET 4.5.2 assemblies (and one of the other projects). In the other two projects I reference WebDriver and Netwonsoft.Json.

On the machine (VM) where it's not working I installed NUnit with NUnit.Console.msi (3.8.0). On the machine where it's working (my workstation) I installed NUnit via NuGet, in Visual Studio, and I start the tests within Visual Studio using NUnit3TestAdapter.

I see now that I'm running NUnit 3.9.0 on my workstation. I will upgrade to 3.9.0 on the VM and see if this helps...

---

@niwe commented on Mon Apr 30 2018

Hmm, I think I have mixed it up. I thought nunit-console and nunit was pretty much the same thing. Now I see that version 3.8.0 is the latest as far as nunit-console goes. So I am running the latest versions on both machines.

Any tips on how I can get this working?

---

@ChrisMaddock commented on Mon Apr 30 2018

Try uninstalling the NUnit3Adapter NuGet package, and then running with the NUnit.Console msi on your VM.

I think there may be a discovery issue with the adapter and msi currently using different using different version of the NUnit engine.

---

@cmoonExpedia commented on Tue May 01 2018

Could anyone expert in NUNIT library explain why NUNIT needs to acquire the remote agent even though the tests run in only local? What is going on under the hood?

Thanks.

---

@CharliePoole commented on Tue May 01 2018

In this case "remote" merely means "in another process." NUnit console runs each assembly in a separate process by default. The process has to be started and then connected to in order to issue commands telling it to load and run the tests. The technology used to do this is called ".NET Remoting."

---

@WereDev commented on Thu May 10 2018

I recently started getting this issue as well. It happens with all solutions/assemblies. I've tried uninstalling/reinstalling NUnit which didn't help, though don't much expect it to given that it's using the one pulled down from NuGet local to the solution. Tried deleting those files to have them pulled down again, but that didn't help either. Again, not really surprised as this suddenly affected all of my solutions at once. I'm not sure what might have changed on my system to cause the issue.

---

@ChrisMaddock commented on Thu May 10 2018

@WereDev do you have nunit installed via the msi, on anything nunit-y on you path vars- in addition to our nuget packages?

---

@WereDev commented on Thu May 10 2018

Had it installed (and reinstalled) via the MSI. Actually have it uninstalled now as the scripts seem to be running console from the nuget packages. Nothing in the environment vars. Haven't checked the registry yet, but I'm unsure what to look for there.

---

@niwe commented on Thu May 24 2018

I finally found a solution to my issue. Hopefully someone can I explain why this is happening.

I ran my tests using (which gave an exception):
C:\PROGRA~1\NUnit.org\nunit-console\nunit3-console.exe C:\...\GUITests.dll

But running it this way works:
"C:\Program Files\NUnit.org\nunit-console\nunit3-console.exe" C:\...\GUITests.dll

Don't know it doesn't work the original way. No big deal for me, I found a workaround so I'm happy. Maybe it helps someone else running into the same problem.

FYI
The reason why I use PROGRA~1 instead of Program Files is (I think... it was a long time ago :/) because I couldn't install NUnit (MSI package), using the flag TARGETDIR, without using the old DOS 8.3 name. And then I reused that path variable when starting NUnit.

---

@rprouse commented on Wed May 30 2018

@niwe what version of the Console do you have installed?

---

@niwe commented on Thu May 31 2018

@rprouse I installed NUnit using NUnit.Console-3.8.0.msi (from https://github.com/nunit/nunit-console/releases)

---

@Hu3bl commented on Thu Jun 21 2018

We see the same issue on our CI server (Windows Server 2012R2). Occasionally, we get the error

NUnit.Engine.NUnitEngineException : Unable to acquire remote process agent --NUnitEngineException at NUnit.Engine.Runners.ProcessRunner.CreateAgentAndRunner() at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
for every test DLL in a larger project. While looking into it, we discovered that the nunit-agent.exe processes that get launched all crash before connecting to the runner and the Windows Event Log has the following to say about that:

Application: nunit-agent.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ArgumentException at System.Diagnostics.Process.GetProcessById(Int32, System.String) at System.Diagnostics.Process.GetProcessById(Int32) at NUnit.Agent.NUnitTestAgent.Main(System.String[])

We suspect that something is amiss with the command line that is passed to the agent process, but cannot yet prove it.

EDIT: We checked the log files of nunit and everything looks right in the command line passed to agent.
Furthermore, we found the issue #117 that describes a similiar behaviour of nunit. The described workaround to run nunit --inprocess does not work for us since we don't have a 32bit process. However, adding --agents=1 results in running tests, but really slow.

When checking the output of nunit without the --agents=1 option, we see that at least one process finds some tests in a test DLL, but every other test DLLs show the exception we described above. We pass a complete solution file including all tests projects to the nunit console.
As said above, we're using Windows Server 2012 R2 with .NET Runtime 4.5.1, nunit console 3.8.0 and our test projects use nunit framework 3.10.1.

---

@ChrisMaddock commented on Fri Jun 22 2018

@Hu3bl This is an interesting stack trace...but I'm unsure how this could occur! Are you able to share your full logs, including the agent log?

---

@Hu3bl commented on Mon Jun 25 2018

Of course I'm allowed to share the logs with you, however, I had to replace all assembly names by xxx because I'm not allowed to publish them.
I called nunit with
nunit3-console.exe build\xxx.sln --result:TestResult.xml;format=nunit3 --trace=Verbose
The resulting files are attached as zip file.
nunit.zip

---

@ChrisMaddock commented on Mon Jun 25 2018

@Hu3bl Is this all the logs that were written? No nunit-agent_6132.log? 😞

---

@ChrisMaddock commented on Mon Jun 25 2018

Never mind. Based on your stack trace, the agent will be crashing before it initialises the log.

But I'm still not sure why. 🤔 Looking at the Agent code - the line which is crashing could be delayed. The Process.GetProcessById() could be deferred to the point where the process is actually used - and we could hold on to the ID instead, and then just look it up when we need to check if it's exited. (Which is already an 'error-case' in itself!)

Would you be able to give that a try, and see if it fixes your issue, or if this is just hiding a second problem? The file I'm looking at is Program.cs in the nunit-agent project.

---

@ChrisMaddock commented on Mon Jun 25 2018

@nunit/engine-team

Nice self-contained issue here. How could Process.GetProcessById() be throwing an ArgumentException from some agents but not others, when the process it is looking up (appears to be) alive and well? It's only documented to throw this when the process is dead - I had a quick look in ReferenceSource, which seems to suggest the same...

---

@CharliePoole commented on Mon Jun 25 2018

Is it possible that the argument is not being parsed correctly in some cases?

---

@ChrisMaddock commented on Mon Jun 25 2018

I thought that at first, but I can't see how that could be the case. The agents all receive the same command line, and 2/n connect fine. @Hu3bl's stack trace also shows the int.Parse line doesn't throw, but the Process.GetProcessById() does.

---

@CharliePoole commented on Mon Jun 25 2018

Locale? Grasping at straws here!

---

@ChrisMaddock commented on Mon Jun 25 2018

I think we may need to add more logging!

---

@jnm2 commented on Mon Jun 25 2018

and we could hold on to the ID instead, and then just look it up when we need to check if it's exited. (Which is already an 'error-case' in itself!)

Process IDs are recycled. If possible, you want to open the process handle (GetProcessById) ASAP in case the process closes and a new process starts with the same ID.
If there isn't a reliable way to be sure the process is still running even when you call GetProcessById ASAP, the usual technique to uniquely identify a process over time is pid + start time.

---

@ChrisMaddock commented on Mon Jun 25 2018

Good shout!

---

@ChrisMaddock commented on Wed Jun 27 2018

@Hu3bl - this issue really should be in the nunit/nunit-console repo. I'm going to move it over now, and comment further over there. 🙂

[ChrisMaddock]

@Hu3bl - I've done a PR which adds some extra logging over at https://github.com/nunit/nunit-console/pull/435.

You should soon be able to download a build of the console with this additional logging from the artefacts tab of: https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.5.4012

Would you be able to try running with that build, until you encounter your issue again? Then if you could again share the logs, they should hopefully contain a little more information. I'm not 100% sure the additional information will tell us a lot, but I'm running out of other ideas to move forward!

[Hu3bl]

I will test it tomorrow!

[Hu3bl]

I don't know if I did everything correct. I downloaded the .msi artifact from https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.9.0-ci-04011-agent-debug/artifacts (The link you provided says: Build not found or access denied.) and installed it on our CI server.
This time, I received 32 log files. Before renaming all dll names in the log, I checked out the files myself after reviewing your changes in the pull request.

Unfortunately, there is no new log message at all. I searched all files for the newly added log messages, but there are none. The log files of the agents start all with Agent process 1234 starting, which is right after your added lines.

Could you add logging right before calling GetProcessById? Maybe you can try to log agencyProcessId variable after int agencyProcessId = int.Parse(agencyPid);.

EDIT: I forgot to say that the event viewer does not show the System.ArgumentException anymore. Looks to me that the exception is catched, but no log is created.

[ChrisMaddock]

Good idea logging the success as well as the failure - I've added a new commit to the PR.

Can you share the full log of one of the failed agents? Presumably there's more than just that Agent process 1234 starting line? I wonder if it's crashing in a different place to where we expect - as I can't see how it could print that message if it was crashing when obtaining the process handle...

[Hu3bl]

I put all logs except the TestResult.xml into the attached zip file.
Additionally, I made screenshots of the task manager where we can see the PIDs, right immediately after the start of nunit-console and after 30 seconds. These files are also in the zip file.

I wonder, why the parent process (nunit-console) is not available anymore when an agent tries to get the pid of the nunit-console process. Check out the timestamps!
I guess the nunit console process exited before the agents start to do the work. But I don't understand, why --agents=2 does the trick and everything works fine.

nunit-agent-debug.zip

[ChrisMaddock]

Great - thanks! Don't have time to fully dig into this fully right now, but there's definitely a few interesting things going on! I think we're barking up the wrong tree - looks like the agents are correct - and the process indeed doesn't exist at the point where they end up looking for it. I'll have a dig into the engine dispatch code at some point...

I see there's a single x86 assembly - which always seems to be one of the only two that runs. Just a hunch - can you try taking that assembly out of your VS solution, and see if everything starts working?

[CharliePoole]

Can you try your run with various numbers of simultaneous processors, i.e. --agents=n?

Here's my working theory... the engine is launching your 27 agents and waiting for them to report back. There's a timeout for that - IIRC 20 seconds. But it's doing this all on the same thread. Somehow, with too many launched at one time, there may be a point where it crashes. If your tests work well up to some number of agents, that would be evidence that this is the right place to look.

[ChrisMaddock]

@CharliePoole - I think you might be on to something.

We have a handle on the agent process at this point. We could wait for 30 seconds of Processor Time - as opposed to 30 seconds in general.

That should protect us against starvation, although I guess puts us at much more risk of a hang. Perhaps we could tie that in with some monitoring of free CPU availability too...

[Hu3bl]

@ChrisMaddock
I changed the assembly executed as x86, but nothing changes. It was just the "Prefer 32-bit" flag that was accidentally set.

@CharliePoole
I tried a various number of agents, and you're right: up to 8 agents, everything works fine, but all above results in an exiting nunit3-console process before all agents start working. Sometimes, it works with the specified number of agents, but the higher the load on the CI server, the more likely nunit3-console exits before the work is done.

Maybe it is worth a try to make a version where we can specify a timeout from the command line, just to test if it works then.

[ChrisMaddock]

@Hu3bl

Been away for a few days, but this one's been on my mind.

I've added one further commit to the PR, which bases the timeout on the amount of processor time available to the agent process, as opposed to a fixed 30 seconds. This isn't code I think we'd want to merge I don't think (sounds dangerous for an overall process hang!) - _but_ if our suspicions are correct on processor starvation, this should allow your tests to run to completion.

If it does work, then we'd need to look at a better solution. We've had a vague plan to create the test agents 'just-in-time' for a while I think - and this might be a good motivation to work on that.

Will you pull the CI build for the latest commit, and try it out for us? 🙂

[CharliePoole]

@ChrisMaddock Actually we are creating the agents just in time. But in thus case we're asked to run them all in parallel, so all are needed.

[aolszowka]

@Hu3bl How many Cores do you have on this test machine? We've seen a significant slow-down if we attempt to use more than the number of cores assigned to a VM. IIRC the default behavior of NUnit3-Console is to spin up as many agents as test assemblies passed.

Perhaps a reasonable fix would be to Limit the number of agents to Environment.ProcessorCount (https://msdn.microsoft.com/en-us/library/system.environment.processorcount.aspx) by default?

It looks like there is a couple of places this could be done.

The first place that might make sense is changing ConsoleOptions.cs _maxAgents to use the above property. https://github.com/nunit/nunit-console/blob/f98bd132d9bd25330e3cbf996571414b68b582b6/src/NUnitConsole/nunit3-console/ConsoleOptions.cs#L82

I think this is probably the "correct" solution.

Another option would be to change MulitpleTestProcessRunner.cs https://github.com/nunit/nunit-console/blob/d72b92e4fab59333f6636330054820d96a01d9e7/src/NUnitEngine/nunit.engine/Runners/MultipleTestProcessRunner.cs#L49

However that seems too low level to me.

[ChrisMaddock]

@CharliePoole - whoops, sorry! Getting ahead of myself...

@aolszowka - exactly what I was thinking! 😄 We already do this for the NumberOfWorker threads - it seems odd that threads default to the processor count, but agents defaults to int.MaxValue. @CharliePoole - is there anything I'm missing there?

My only comment above what you've suggested is that this is a fix that belongs in the engine, in my opinion. (i.e. the MultipleTestProcessRunner.) My reasoning being that the engine is designed used by many different 'runner-programs' - the console being one, but also the VS adapter, the GUI Charlie is working on, and various third party projects. This feels like an issue which is bite any of these runners, and therefore we should give a more sensible default on the engine.

If @CharliePoole agree's with what we've discussed, would you be interested in doing a Pull Request for this? 🙂

[CharliePoole]

Two reasons, one rooted in the past, one in the future!

1. Multiple process execution was there first, before we added the ability to limit the number of agents.

2. We envisioned running the processes on separate machines eventually but never got there.

Note that if you use multiple processes AND multiple worker threads, the default load would end up being the square of the number of cores.

[ChrisMaddock]

Thanks for the background!

Note that if you use multiple processes AND multiple worker threads, the default load would end up being the square of the number of cores.

Yes...I thought that, but wasn't too sure how best to limit it. 🤔 I don't think we can limit MaxAgents based on Number of Worker thread, as _unless_ NumWorkers is fixed by a PackageSetting, the LevelOfParallilization is unknown until runtime, based on any parallelization attributes the tests may be using. Therefore, in a 4-core system, we may have MaxWorkers as 4, but only one worker actually used, as the tests are run lineally...

Any ideas?

[ChrisMaddock]

I guess if NumAssemblys > NumProcessors, we could add MaxWorkers=1 to every package, and the maximise the parallelisation in terms of agents instead.

That's just potentially a slow down if you have many small test assemblies, and a couple of big ones - which is definitely a common scenario in my life.

[CharliePoole]

It never seemed that there was something we could do "for the user" that would not hurt some users. The --agents setting was added so that users could optimize it themselves. Note that the old behavior of NUnit V2 can be acheived by setting Agents=1.

My general assumption has been that the vast majority of users with simple applications have one test assembly. I have nothing to back that up. 😊

[Hu3bl]

@ChrisMaddock the new version you provided does not change anything. Still the same behaviour like before.
Our CI server is a virual machine with 4 cores attached. I had no issues the last days running nunit with --agents=4.

[ChrisMaddock]

@Hu3bl Just looked at my code again - there's a bug, sorry - it won't be testing what I was hoping to test.

Will try and get it fixed up some point soon.

[ChrisMaddock]

@Hu3bl New attempt, without bug pushed to the PR.

Glad --agents=4 is working out for you! If you'd be happy to continue helping, I'd quite like to double check our suspicions as to the cause are right, so we can fix this in the 'default set-up', and hopefully others won't hit the same problems!

[Hu3bl]

@ChrisMaddock I just tested your fixed version and well, it is working for us without any --agents=x restriction.

Of course I will continue helping since this makes nunit more stable in different application areas on different hardware. Just let me know when you have new versions to test.

[ChrisMaddock]

Great - thanks!

So - I’d expect using the —agents option will allow your tests to run faster. But it’s good that we’ve found a solution that doesn’t crash in general!

I suggest we add a better default on MaxAgents, and also update the agent registration code to not fail under high processor load. I’m not totally sure about the best approach to the latter bit - I’m a bit concerned my current solution could lead to hangs under certain situations. I’ll have more of a think - maybe a combination of testing for process liveness and cpu time...

[Hu3bl]

Using --agents=4 does not make a big difference compared to run nunit without a restriction when it comes to execution time. Our tests run approximately 20 minutes in both cases. There is only a difference of few seconds.
Nevertheless, memory usage is higher without the restriction, but memory is not the bottleneck while executing the tests. It is the cpu.

[ChrisMaddock]

That's interesting! I'd have thought the resource contention may have slowed things down a little. I guess it must be a little machine dependent - @aolszowka said that this case caused a noticeable slow down for them.

It's good to know that setting max agents doesn't slow things down further compared to allowing unlimited, which means that changing the default value to the number of cores hopefully won't have a detrimental effect on other users. I guess we need to remember that our defaults are intended to provide the best out-of-box experience, but people may still want to fine tune for the most optimal behaviour...

[Hu3bl]

I tested another project with a much longer execution time for the tests. With the restriction to four agents, it ran 2h 5min, without restriction 2h 14min.

Like @aolszowka already mentioned, it slows down notriceable, but it is kinda project dependent. For the project we had the issues with we did not really notice a difference (only a few seconds).

[ChrisMaddock]

Cool, thanks. 🙂 I remember you stated your original project doesn't use any parallelization within assemblys - is this second one the same? I'd imagine that could make a difference. (e.g. 4/5 times the number of competing threads!)

Sounds like it's is a good plan for us to change the default though - thanks for looking into it further!

[Hu3bl]

The second project and especially its tests use a lot of I/O stuff. I guess this could be an issue, but I'm not sure. We don't use parallelization in there.

[BADF00D]

As I mentioned in #117, I'm using NUnit 3.8.0 with OpenCover 4.6.519..
I made some test with my solution and I want to share the results:
I have 29 assemblies with tests and a machine with 4 logical cores and 24GB RAM. Limiting the number of agents to 2 or for solved the problem for me. Afterwards I started increasing the number of agents.
Interestingly, I was able increase the number to 8 and 10, without any problems.

Then I set the limit to 29, and the problem reappears. I got the error "unable to acquire..." from 28 processes, but I got some result from 8 assemblies. That sounds strange to me. So I started some more tests runs:

| Agend Limit | Errors² | Assemblies³ | Covered lines³ | Uncovered Lines³ | Expectation meet |
|---|---|---|---|---|---|
| 2 | 0 | 29 |22033|4568|✔|
| 20 | 0 | 29 |22032|4568|✔|
| 21 | 0 | 29 |22033|4568|✔|
| 22 | 0 | 29 |?|?|✔|
| 23 | 0 | 29 |?|?|✔|
| 24 | 0, ? | 29, 28 |?, 21253|?, 4858|✔, ✖Number of covered lines went down|
| 25 | 0, ? | 29, 28 |22016, 14663|4585, 11299|✖Number of covered lines went down|
| 26 | 21 | 28 |?|?|✖|
| 27 | 22, 26 | 11, 6 |?|?|✖|
| 28 | 26 | 8 |?|?|✖|
| 29 | 28, 28 | 5, 8 |?|?|✖|

• ² Number of process that report "Unable to acquire... ". Sometimes I ran the configuration twice.
• ³ Number of assembly/uncovered lines/covered lines that appear in OpenCover report (they were sometimes incomplete and coverage is wrong)

I ran each configuration at least 2 times, sometimes with different result.

My conclusion:

1. I'm not sure if there is a connection between number of cores and the problem.
2. I'm not sure if this an initialization problem. Sometimes I got the error from a process, that also performed some tests.

By the way, I could not see any problems, when running all test without OpenCover.

[ChrisMaddock]

@BADF00D - OpenCover adds a whole new variable, I personally know nothing about how it works! You could well be seeing the same exception with a different cause. 🙂

Would you try out the test fix we have here, from PR build at the link below - and see if that allows you to complete the run with no --maxagents limit?

https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.9.0-ci-04018-pr-435/artifacts

[ChrisMaddock]

@BADF00D Did you manage to try the test build? I'd like to come back to this issue and tie it up at some point - would be great to check out if this also solves your problem before that point.

[BADF00D]

Hi,
@ChrisMaddock I know that OpenCover introduces a whole new variable. But in this case, I was able to reproduce the problem and therefore I want to share what I found out.

I just tested the new version 3.9.0.0 from 26th march 2018.

| Agend Limit | Errors² | Assemblies³ | Covered lines³ | Uncovered Lines³ | Expectation meet |
|---|---|---|---|---|---|
|29 | 28 |5 |829|2743|✖|
|not set| 28 |4|2095|3521|✖|
|1| 0 | 29| 22032| 4568|✔|
|2| 0 | 29| 22032| 4568|✔|
|8| 0 | 29| 22058| 4543|✔|
|24| 9 | 29| 22033| 4568|✔|
|29| ? |5| 2918| 3632|✖ - 26 Agents not terminated|

Btw. when the error obscures, sometimes some NUnit Agent process are still running, after NUnit Console Runner terminated (see last test).

[ChrisMaddock]

Hey @BADF00D - sorry, I think we're mis-communicating!

I wouldn't expect this error to be fixed in 3.9.0, but we're working on something which I think might fix this in a pull request, which hasn't yet been merged into the main codebase, and isn't yet released.

You can download the pre-release version with the fix applied from the link below. If you'd be able to try your tests with that version, that would be really helpful - that way we can see if our fix for the issues @Hu3bl is seeing also fix your issues.

Download link: https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.9.0-ci-04015-agent-debug/artifacts

[aolszowka]

So looking at the OpenCover Source it appears they just execute the console application with arguments and expect a non-zero exit code: https://github.com/OpenCover/opencover/blob/19ebe9c19ba0f8205f28b6923f7c7d5e2730004a/main/OpenCover.Console/Program.cs#L346 however this is deceptive as it has to hook the profiler into any of the processes, it appears most of the heavy lifting is done here https://github.com/OpenCover/opencover/blob/530afa03a5963548e4a47fe82324b582a81c9e3b/main/OpenCover.Framework/Manager/ProfilerManager.cs#L26

It was this comment that interested me from you above:

when the error obscures, sometimes some NUnit Agent process are still running, after NUnit Console Runner terminated (see last test).

As you've deduced you're probably seeing additional reports because for some reason OpenCover is able to attach to those still running Agents.

I think to narrow the report down a bit is we need to find in what scenarios NUnit-Console will exit while still allowing the NUnit-Agent processes to exist, that's probably your root cause (and will get the OpenCover variable out of the picture).

[ChrisMaddock]

@aolszowka Thanks for digging into that! Really useful to get a bit of an insight into what Opencover is doing.

I think to narrow the report down a bit is we need to find in what scenarios NUnit-Console will exit while still allowing the NUnit-Agent processes to exist, that's probably your root cause (and will get the OpenCover variable out of the picture).

That's hopefully just what this issue is! 🙂 The differences - this issue, we think was caused by CPU starvation, by having too many agents running simultaneously. My guess with Opencover is that it's also CPU starvation, as I imagine opencover is fairly resource intensive, and doing a lot of work and taking up plenty of CPU cycles while the tests are running.

I'm hopefully the fix in the PR will fix both of these issues, but it would be great if @BADF00D would be able to test that out with her/his scenario, before we work out how to implement the fix properly! 🙂