Nunit-console: SocketException thrown by nunit3-console.exe --explore option
@JackUkleja commented on Wed Oct 12 2016
I am running NUnit.ConsoleRunner.3.5.0\toolsnunit3-console.exe with the --explore option against one of my test assemblies and I am getting the following stack trace:
System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestAgent.Stop()
at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.MasterTestRunner.Dispose()
at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter)
at NUnit.ConsoleRunner.Program.Main(String[] args)
Any clues what is going on? The tests in this assembly run just fine, but the explore options seem to kill the runner.
@ChrisMaddock commented on Wed Oct 12 2016
@JackUkleja - What error do you see if you add the --inprocess option?
@JackUkleja commented on Wed Oct 12 2016
If I try --inprocess I get:
NUnit.Engine.NUnitEngineException: Cannot run tests in process - a 32 bit process is required.
So obviously my test DLL is 32 bit and the nunit-console.exe is running as 64 bit due to it being 64 bit OS.
If I use Corflags to force nunit-console.exe to run as 32 bit then the --explore option appears to work just fine.
So it seems I am unable to replicate the original exception when doing --inprocess.
@CharliePoole commented on Wed Oct 12 2016
It's useful info nonetheless. We should see if we can replicate this.
@ChrisMaddock commented on Thu Oct 13 2016
Hmm...I can't reproduce this with a simple x86 dll.
@JackUkleja - What's your entire command line, is there anything else involved? Could you try setting up a minimal example, with a 1 test assembly you'd be able to share?
@JackUkleja commented on Mon Oct 24 2016
One more clue about this...
I was just trying to replicate this issue with the following command:
"NUnit\NUnit.ConsoleRunner.3.5.0\toolsnunit3-console.exe --explore Artefacts\Debug\Tests\IntegrationTests.dll"
and much to my surprise it worked without any errors. I got a list back of the contained tests, as expected! So I tried a few of my other test assemblies - they all worked too. Finally I tried my IntegrationTests.dll once more, exactly the same command line,...but this time it failed with the aforementioned SocketException
@JackUkleja commented on Mon Oct 24 2016
Nothing I can do now seems to let me run this command successfully again. I've rebuilt the assemblies, restarted the console, looked for any "remoting" or "nunit" processes that maybe could be killed or restarted (nothing). Is something persisting in the remoting infrastructure that is breaking things - something that is not being cleaned up? I'll try log off/on to see if that gets it working again....
@JackUkleja commented on Mon Oct 24 2016
Oh and this only seems to be happening with my IntegrationTests.dll so there is clearly some strange specific interaction between that dll and nunit remoting.
Update: The command just randomly succeeded without me changing anything specifically. I just ran it a bunch of times and about once in 50 attempts it worked again...
@rprouse commented on Thu Dec 01 2016
@JackUkleja do you have any updates on this? I suspect that it is something in your tests and would like to close this issue.
@tfabris commented on Wed Mar 22 2017
Our company is getting this exact same error with this exact same stack trace. It is rare and intermittent for us. We have unit tests that run several thousand test cases on several different code branches several times per day, And we're hitting this perhaps twice per day overall. So it's quite rare for us. But we do hit it. So rprouse, please don't close the issue just yet.
We are running Nunit version 3.6.1 by the way.
@tfabris commented on Wed Mar 22 2017
To clarify: When it happens, it's random as to which test case happens to be running when it hits. So we don't think it's related to something specific in any of our tests. 99% of the time the tests run just fine and they all pass, it's only on the rare occasion that we hit this error, and when we do, it's randomly somewhere in the test list, not the same place every time. (But when the crash occurs, it's always the exact same stack trace shown at the top of this thread.)
@rwencel commented on Thu Mar 23 2017
We have the same problem, same description, though we don't use the explore option, we just run the tests in parallel. 100% of tests succeed, but at the end of the run we encounter this error. It's sporadic. One theory is that it's when our build server is under stress (multiple builds running at once), but I haven't confirmed that.
I'm trying a build with the workaround mentioned by @eberlid in the issue below to see if it fixes. I'm not sure it's a proper fix b/c it just ignores the exception, but it seems harmless and I'm not sure what the root cause is.
NUnit Console Runner Issue #171
@tfabris commented on Thu Mar 30 2017
This issue continues to occur for us intermittently when using Nunit 3.6.1 to run our unit test and integration test suites on Team City.
Because it's so intermittent, we don't have a way to reliably reproduce the issue locally on our local machines. Is there any forensic information we can obtain from Team City that would be helpful?
It would be nice to be rid of these "exit code -100" socket exception errors, since they block production when they occur.
@ChrisMaddock commented on Fri Mar 31 2017
I also found one of our assemblies which hits this yesterday - but couldn't for the life of me manage to debug it. 😞
It feels like a race condition - as I can't reproduce it when running through visual studio or in debug mode. Logging makes it harder to come by - I have some logs, but they're not useful.
The only difference I can really think about in this assembly which is erroring is that it's far more dependent on testcasesources than some of our other assemblies. At some point, I'll try and test out that theory with some slow testcasesources.
@jnm2 commented on Fri Mar 31 2017
I've hit this in debugging a fair number of times a few months ago but none recently.
@ChrisMaddock commented on Fri Mar 31 2017
For what it's worth - this is what the logs look like up until the crash, with --trace=Debug. I was running a single assembly, with just the --explore option.
Engine
13:50:38.052 Debug [ 1] TestAgency: Waiting for agent {7c42103a-6371-4f5f-ba31-df7e7016a6a6} to register
13:50:38.253 Debug [ 1] TestAgency: Returning new agent {7c42103a-6371-4f5f-ba31-df7e7016a6a6}
13:50:41.227 Info [ 1] ProcessRunner: Unloading
13:50:41.313 Debug [ 1] ProcessRunner: Stopping remote agent
13:50:41.321 Error [ 1] ProcessRunner: Failed to stop the remote agent. An existing connection was forcibly closed by the remote host
(log continues - stops all services as expected)
Agent
13:50:38.294 Info [ 5] NUnitFrameworkDriver: Loading Tests.exe - see separate log file
13:50:39.956 Info [ 5] NUnitFrameworkDriver: Loaded Tests.exe
13:50:39.957 Info [ 5] NUnitFrameworkDriver: Exploring Tests.exe - see separate log file
13:50:41.315 Info [ 5] RemoteTestAgent: Stopping
(End of agent log)
@tfabris commented on Mon Apr 03 2017
@rprouse and @CharliePoole , do you guys have any idea about where this is coming from? It only started occurring when we upgraded from NUnit 3.4.1 to NUnit 3.6.1, so it is definitely a new regression which occurred somewhere in between those two builds.
@CharliePoole commented on Mon Apr 03 2017
@ChrisMaddock Didn't you confirm this already?
@ChrisMaddock commented on Mon Apr 03 2017
@CharliePoole - yes, although only as in so much as I can see it happening, I've not got much idea what's wrong!
I was hoping to take some of the characteristics of the assembly I see it happen with a lot, and create a repro I can share. Haven't managed to get back to that yet.
Update: Nope...not what I thought unfortunately...
@eberlid commented on Mon Apr 03 2017
We got the same symptoms when we began to start multiple Gallio test
runners in parallel using MbUnit test framework. Before that we did not
start multiple runners in parallel and we did not get this issue.
Probably both runners have something in common?
See https://github.com/nunit/nunit-console/issues/31#issuecomment-267059306
@CharliePoole commented on Mon Apr 03 2017
@eberlid Insofar as I understand this issue is about running a single copy of the console runner using --explore option.
I can envision all sorts of problems that may come up running multiple copies of the runner at the same time and we should actually have another issue for that. Each instance of the engine starts up a server to which the agents launched by that instance report. So with n consoles and n agents running, there's lots of room for unexpected stuff to happen, especially as this was not a use case we originally had in mind.
@tfabris commented on Mon Apr 03 2017
@CharliePoole, our repros are all running a single copy of the console runner on any given computer as far as I know. We're launching these tests from within a single build step in Team City, and we're not yet taking advantage of the parallelization features built into Nunit 3 yet.
However, we do have a fleet of multiple test agent computers on the network. So more than one testrun might be executing simultaneously on completely different computers. I'm assuming that by "single copy of the console runner", you mean, a single copy on that particular computer.
@CharliePoole commented on Mon Apr 03 2017
@tfabris Yes, that's what I meant - i misunderstood what you were saying.
@tfabris commented on Mon Apr 03 2017
Well, maybe our situation is different than @eberlid. I don't know how MbUnit runs things.
@CharliePoole commented on Mon Apr 03 2017
@tfabris Sorry to confuse... my comment was meant for @eberlid
@JackUkleja commented on Mon Apr 03 2017
@CharliePoole OP here. Just FYI my gut feeling is that --explore option is probably not salient to this bug, but I have not experienced the issue since originally posting and have not personally seen it during normal test runs.
It is interesting this thread had a spate of repros in the last couple of weeks. Does this coincide with a new release? Or perhaps this issue is showing up more in organic search? I'm just surprised at the sudden influx of repro after 3 or 4 months of silence.
@tfabris commented on Mon Apr 03 2017
@JackUkleja, that's probably my fault. The spate probably started when I came a-posting a couple of weeks ago. In my particular case, it was because we had been on NUnit 3.4.1 until then. We upgraded to NUnit 3.6.1 on our Team City server, and that's when suddenly we started seeing our Team City builds getting all these "the process exited with code -100" errors. Then I did some searching and came across this thread showing the exact same stack trace, and posted about it,, which woke things up again for this thread.
We run thousands of unit tests and integration tests every day, and we encounter this error a few times each day. So that should give you an idea of the frequency of the problem, though I don't have exact numbers.
Even though that seems like a rare frequency for this crash, it has an interesting effect on us because the tests are a big part of our production chain. When the tests fail with this crash, it's brings our production grinding to a halt and we have to re-run everything from the top of the chain again. So we're keen on finding either a solution or at least a work-around.
@kditrj2d commented on Mon Apr 17 2017
Hello, I am another user who is encountering this exact same exception and stack-trace. For me the issue has been happening for a while. It was rare (once a day or so) when we were running NUnit 2.6 tests with the NUnit 3.6.1 console runner. We have just upgraded our tests to NUnit 3.6.1 and the problem has become more serious, so much so that it is blocking our CI build. We are currently running CI from within Team City. The last test assembly reported before the crash runs perfectly happily on the command-line and from within VS2015 via the R# test runner. I haven't tried running with --inprocess yet. I don't believe that any of our tests are currently using any of the parallelism features available in NUnit 3.
@baluMallisetty commented on Tue Apr 18 2017
Hello, Please check if there is any TCP connection being established along a NAT channel. .net has remoting issue with TCP over NAT. Use WCF if possible.
@tfabris commented on Mon May 08 2017
@CharliePoole any word on this issue? We continue to encounter it randomly among our thousands of unit and integration tests, when running Nunit 3.6.1.
It only seems to occur when we have the tests running on Team City agent servers. I don't think any of our developers has reported it happening when they run tests on their local computers. I'm not sure if that's due to the special command line parameters that Team City uses to launch the tests, or if it's merely a quantity thing (our Team City servers are where the tests get run most frequently).
@tom-dudley commented on Mon May 08 2017
We've also been seeing the same exact message when running via Teamcity, using 3.6.0. The arguments being used are:
nunit3-console.exe foo.nunit --result=bar.nunit.xml --noheader --framework=net-4.0 --where "cat != SomeTests" --explore=baz_all.tests
We're currently avoiding the issue by using --inprocess
@tfabris commented on Mon May 08 2017
@tom-dudley - Thanks for the tip, I will experiment with adding "--inprocess" as a temporary workaround and see if it solves the problem for us.
If I understand it correctly, this prevents parallelism for test fixtures which support parallelism? So it would only be a useful workaround for folks who don't use the parallel features. Am I correct in that understanding?
@tfabris commented on Mon May 08 2017
@tom-dudley - I take that back. I'm not able to use "--inprocess" as a workaround because some of our test cases must be run with the --x86 parameter, and if I try to add "--inprocess" I get an error message saying that the two parameters are incompatible.
@CharliePoole - Note: Possible help to narrow this down... At least one of the places where we were getting this socketException error is specifically in our 32-bit tests which were running with the --x86 parameter. I don't have data on whether all instances of the socketException that we encountered were "--x86" cases or not.
@tom-dudley commented on Mon May 08 2017
I managed to strike lucky and find an assembly where this is mostly reproducible. Everything in the Dispose looks to be working fine up to calling _agent.Stop() in ProcessRunner. Sticking in a call to the agent in the line before works, so the connection seems fine at this point. We then step over to the RemoteTestAgent.Stop code and execute stopSignal.Set(). After that line executes we immediately hit error handling back in ProcessRunner and bubble it up. So I wonder if this ManualResetEvent is causing us to exit prematurely before we're actually done with the object. The termination logic for the agent is here.
I can't see any obvious reason why this might fail, but someone else might spot something. If anyone has any other ideas for things for me to try with this assembly I'm all ears. I'll see about putting together a minimal one if the issues continues on.
One thing of note however was that if I run nunit3-console without the framework assembly in the same directory, my test assembly fails 99% of the time. But if I drop the framework assembly in the folder then everything works. Not sure what's going on here! (I'm checked out at 3.6.1 for this work)
@tfabris commented on Mon May 08 2017
@tom-dudley and @CharliePoole - If it helps, I did some history digging and found that the socketException has occurred randomly on both our regular 64-bit tests as well as our 32-bit tests that were run with the --x86 command line parameter. So it's not related to that.
@rprouse commented on Mon May 08 2017
@tom-dudley I also don't see anything obvious. Since you are able to fairly reliably reproduce this, what about wrapping the stopSignal.Set() and the stopSignal.WaitOne(timeout) in try/catch to see if it is one of these that is throwing and if so, see if you can get more info out of the exception. Maybe even do the same for the while loop in WaitForStop()?
I am hoping that we can start narrowing down exactly what is throwing the exception that seems to be killing the agent.
@rprouse commented on Mon May 08 2017
@CharliePoole and @ChrisMaddock is there any reason we haven't moved this to the NUnit Console repo? Is it only because it has the confirm label?
@rprouse commented on Mon May 08 2017
@CharliePoole, just curious why we use a TcpChannel rather than an IpcChannel. I would think that an IpcChannel would be more reliable and faster since we never run agents on other machines. I don't think it would fix this issue, but I am curious about the history. Was it because we intended to run agents on other machines? If so, should we rethink that until it is needed?
@tom-dudley commented on Tue May 09 2017
Turns out the RemoteTestAgent is exiting cleanly as far as it is concerned. It always hits return 0 in Main.
I believe the following code may reliably reproduce the error. I still don't understand why this causes the exception to be thrown though!
public class Tests
{
public static readonly string[] TestDirectory = { TestContext.CurrentContext.TestDirectory };
[Test]
public void MyTest([ValueSource(nameof(TestDirectory))] string directory)
{
}
}
@ChrisMaddock commented on Tue May 09 2017
I also have an assembly which produces this fairly reliably with --explore - but I haven't been able to create anything reproducible. 🙁 Anything I've done to add logs, or try and catch the exception, seems to suppress the error - I guess by slowing the process down.
is there any reason we haven't moved this to the NUnit Console repo?
I only haven't moved it as we haven't tracked down anything to actually fix yet. It's also likely a duplicate of some of the below - but I didn't think we could really confirm that, till we had something to test... 🙁
https://github.com/nunit/nunit-console/issues/171
https://github.com/nunit/nunit/issues/2027
https://github.com/nunit/nunit-console/issues/219
@jnm2 commented on Tue May 09 2017
I can manage to get a socket exception once in a while by repeatedly running .\nunit3-console.exe mock-assembly.dll mock-assembly.dll mock-assembly.dll [...] --process=multiple against 3.6.1:
Unhandled Exception: System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.Tcp.TcpFixedLengthReadingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BinaryReader.ReadBytes(Int32 count)
at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryResponseMessage(Stream inputStream, IMethodCallMessage reqMsg, Boolean bStrictBinding)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestAgent.Stop()
at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing) in C:\Users\Joseph\Source\Repos\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:line 258
at NUnit.Engine.Runners.AbstractTestRunner.Dispose() in C:\Users\Joseph\Source\Repos\nunit-console\src\NUnitEngine\nunit.engine\Runners\AbstractTestRunner.cs:line 225
at NUnit.Engine.Runners.TestExecutionTask.Execute() in C:\Users\Joseph\Source\Repos\nunit-console\src\NUnitEngine\nunit.engine\Runners\TestExecutionTask.cs:line 47
at NUnit.Engine.Runners.ParallelTaskWorkerPool.ProcessTasksProc() in C:\Users\Joseph\Source\Repos\nunit-console\src\NUnitEngine\nunit.engine\Runners\ParallelTaskWorkerPool.cs:line 81
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
@jnm2 commented on Tue May 09 2017
If someone has a consistent repro, would be fun to play with that.
@tom-dudley commented on Tue May 09 2017
@jnm2 The snippet I posted above is consistent for me. I just realised when I said it didn't show the error any more I was running the wrong assembly. Doh!
@jnm2 commented on Tue May 09 2017
@tom-dudley And is this without TeamCity?
@tom-dudley commented on Tue May 09 2017
Yep, this is creating a new project with that snippet, using nunit-console tagged at 3.6.1 and running it locally with nunit3-console.exe .\NUnitFailure\bin\Debug\NUnitFailure.dll --explore
@jnm2 commented on Tue May 09 2017
My debug and release builds at 3.6.1 don't repro this.
Ah, repro'd with https://github.com/nunit/nunit-console/releases/download/3.6.1/NUnit.ConsoleRunner.3.6.1.nupkg. 🎉
@tom-dudley commented on Tue May 09 2017
Yes I repro'd with both that nuget package and with building nunit-console from source at the 3.6.1 tag. I've not tried the zip or msi on the Releases page of nunit-console.
@tom-dudley commented on Tue May 09 2017
I have an assembly which contains a solitary reference to nunit.framework and it still gives the error.
@ChrisMaddock commented on Tue May 09 2017
This repros for me too. Awesome @tom-dudley, I haven't ever managed to break something out that did! 😄
My assembly I regularly see this with doesn't use TestContext (It mostly pre-dates it!). It does however make extensive use of ValueSources.
@jnm2 commented on Tue May 09 2017
Also repros with:
```c#
public class Tests
{
public static readonly string[] TestDirectory = { "" };
static Tests()
{
_ = TestContext.CurrentContext;
}
[Test]
public void MyTest([ValueSource(nameof(TestDirectory))] string directory)
{
}
}
---
@baluMallisetty commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300227945)
Hello guys,
I don't think I should in this conversations, I believe my email is ccied
by accident, could you please loop me out?
I am not aware of what Nunit is too! I think I am not the right person
included in your email! 😂
Thanks
Balu
On Tue, May 9, 2017 at 12:43 PM Chris Maddock <[email protected]>
wrote:
> This repros for me too. Awesome @tom-dudley
> <https://github.com/tom-dudley>, I haven't ever managed to break
> something out that did! 😄
>
> My assembly I regularly see this with doesn't use TestContext (It mostly
> pre-dates it!). It does however make extensive use of ValueSources.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/nunit/nunit/issues/1834#issuecomment-300226503>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ALrPmS4rfmJ3KsajabZ34yhEOih4VZAKks5r4JebgaJpZM4KUc6I>
> .
>
---
@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300228420)
@baluMallisetty We are not in control of that. You can visit this page (https://github.com/nunit/nunit/issues/1834) and click Unsubscribe in the sidebar for this thread.
---
@jnm2 commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300228647)
Also repros with:
```c#
public class Tests
{
public static readonly string[] TestDirectory = { "" };
static Tests()
{
_ = TestContext.CurrentContext;
}
[TestCaseSource(nameof(TestDirectory))]
public void MyTest(string directory)
{
}
}
@jnm2 commented on Tue May 09 2017
My guess is that the trigger is when NUnit needs to invoke a member in order to enumerate the tests and the class containing the member has a static constructor that accesses TestContext.CurrentContext.
@jnm2 commented on Tue May 09 2017
It's not specifically the constructor, it's just caused by the TestContext.CurrentContext being accessed when NUnit invokes a member in order to enumerate test cases.
```c#
public class Tests
{
public static string[] TestDirectory()
{
_ = TestContext.CurrentContext;
return new[] { "" };
}
[TestCaseSource(nameof(TestDirectory))]
public void MyTest(string directory)
{
}
}
Gotta take a break for a while.
---
@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300236181)
Just to remove all usages of strings:
public class Tests
{
public static readonly TestContext context;
static Tests()
{
context = TestContext.CurrentContext;
}
[TestCaseSource(nameof(context))]
public void MyTest(TestContext context)
{
}
}
---
@tom-dudley commented on [Tue May 09 2017](https://github.com/nunit/nunit/issues/1834#issuecomment-300269559)
After a bit of cowboy-debugging:
```
@jnm2 commented on Tue May 09 2017
@tom-dudley That's because you did public static readonly TestContext context; instead of TestContext[]. It should be an enumerable containing one item per test case.
@tom-dudley commented on Tue May 09 2017
@jnm2 Good point! After stepping into the TestCaseSourceAttribute everything looked fine and I got some sensible XML back. Then I hit the _agent.Stop(), the agent gets to existing its Main and we get the exception.
@jnm2 commented on Wed May 10 2017
Ah, it's more complex.
Right now I can consistently repro if and only if the test assembly has nunit.framework.dll in the same folder and nunit3-console.exe does not have nunit.framework.dll in the same folder.
@jnm2 commented on Wed May 10 2017
If this is accurate, we will not be able to do tests on this scenario without build.cake running nunit3-console.exe from a separate folder. I do think tests on this are important.
@jnm2 commented on Wed May 10 2017
Hey, I'm getting the repro to happen intermittently in a modified build.cake which copies nunit3-console.exe to its own folder before running mock-assembly tests. This is on a fresh debug build of nunit-console.
@tom-dudley commented on Wed May 10 2017
@jnm2 Regarding the non-presense of nunit.framework.dll, I got exactly the same results as you - i.e. nunit-console.exe not having nunit-framework.dll in the same folder causing it to repro
@tfabris commented on Wed May 10 2017
Hm. Is it possible that the issue could be caused by a mismatched version of nunit-framework.dll?
I notice that our tests, when run on Team City, which repro the issue only randomly/intermittently, seem to have the 3.4.1 version of nunit-framework.dll on the hard disk in the folder along with the test assembly. This must be a leftover in the build system that we haven't corrected yet. The tests get run with the 3.6.1 console runner, but there is a 3.4.1 version of nunit-framework in the folder with the tests.
I'm going to try getting our build system to update to the 3.6.1 version of the framework.
@jnm2 commented on Wed May 10 2017
I'm pretty sure the console runner acts differently depending whether the framework DLL is present, and perhaps it's version related, but it should not cause this exception no matter what.
@jnm2 commented on Wed May 10 2017
It turns out to be pretty tricky getting this under automated testing, but it's so worth it.
The test can't be done by an process that references or loads nunit.framework due to the Heisenburg quality of this bug, so the test has to be written in an assembly that does not use nunit.framework. Meaning, no testing framework.
Or, if we're willing to forgo easy debugging, it can be done out of process. That's probably the least disruptive option.
@jnm2 commented on Wed May 10 2017
Hey guys, guess what? I have a reliable test at https://github.com/nunit/nunit-console/pull/223.
@tom-dudley commented on Wed May 10 2017
nunit3-consolewith no framework in the folder. Test linked against 3.6 - Exceptionnunit3-consolewith framework 3.6 in the folder. Test linked against 3.6 - No Exceptionnunit3-consolewith framework 3.7 in the folder. Test linked against 3.6 - Exceptionnunit3-consolewith no framework in the folder. Test linked against 3.7 - Exceptionnunit3-consolewith framework 3.6 in the folder. Test linked against 3.7 - Exceptionnunit3-consolewith assembly 3.7 in the folder. Test linked against 3.7 - No Exception
And by '3.7 in the folder', I just mean changing the assembly version number. All exceptions were repros of the original issue.
@tom-dudley commented on Wed May 10 2017
@jnm2 The tests do a pretty good job at crashing my VS test runner, so looks like they're definitely reliable!
@jnm2 commented on Wed May 10 2017
Have to take a break again, anyone is welcome to help find a fix if they so desire. I'll start looking at a fix when I'm back.
@jnm2 commented on Wed May 10 2017
@tom-dudley Nice! That's valuable information that should probably be tested against too.
@tfabris commented on Wed May 10 2017
This is amazing work you are all doing. Awesome. In the meantime, my boss has updated our build system so that the framework and the console runner match at version 3.6.1, so we'll have additional data about whether it might be related to the version mismatch.
@CharliePoole commented on Wed May 10 2017
I'm joining in late, so may be missing something. However...
It's absolutely certain that the framework you are referencing has to be either in the same folder as your test assembly or somewhere else (like on the probing path) where it can be found. Consequently, there can't be another version of the assembly there, since all versions have the same file name. The framework is an assembly that you - not the runner - reference, so it's up to you to get it there.
In this context "up to you" includes any third party software you are using, like TeamCity or even VS. 😄
OTOH, there is no requirement for the version of the runner to match that of the framework. The runner doesn't reference the framework. Of course, if you are running the nunit3-console runner, your framework had better be >= to 3.0 OR you had better have the V2 framework driver extension. That doesn't sound like your problem here, so I only mention it for completeness.ere can also be a loss of function if the runner is older than the framework - that is, the framework may have features that the runner doesn't understand - but things should still run.
@jnm2 Re Runner acting differently if the framework isn't there... Yes... if the framework is not located with the test assembly that references it, we never find it to load and crash. That may not give the friendliest message and so should be checked.
Guys... why is this an nunit bug rather than a console/engine bug?
@tfabris commented on Mon May 15 2017
I have confirmed that the issue still occurs on our build/CI system, intermittently, even when the framework version and the console runner version are both 3.6.1, and the framework DLL is present in the folder with the test assembly DLL.
@jnm2 commented on Thu May 18 2017
@CharliePoole
Guys... why is this an nunit bug rather than a console/engine bug?
It was never moved. @rprouse asked the same above.
jnm2
All 80 comments
@jnm2 , thank you so much for the detailed analysis and fix in https://github.com/nunit/nunit-console/pull/223 .
Will this issue be updated when a release build is available which contains the fix?
Thanks again for the excellent work.
tfabris
on 26 May 2017
@tfabris the PR has been merged, so this issue is now closed. If you want to test, you should be able to use the appveyor build. Look for 3.7.0-dev-03641 using the NuGet feed https://ci.appveyor.com/nuget/nunit-console and let us know if it fixes your issues.
rprouse
on 26 May 2017
@tfabris We don't update each issue when its fix RTMs, but what you can do is watch https://github.com/nunit/nunit-console/releases (or subscribe to https://github.com/nunit/nunit-console/releases.atom). (And to second @rprouse, please do test the dev build!)
@rprouse Since #223 is in master, I assume I should put all issues fixed by it in the 3.7 milestone.
jnm2
on 29 May 2017
Hi!
We've tried 3.7.0-dev-03644 and it still crash with Socket Exception (0x80004005) under teamcity. Stack trace is the same as before update (v 3.6.1)
demonix
on 2 Jun 2017
Well shucks. Any ideas @jnm2 ?
tfabris
on 2 Jun 2017
@demonix @tfabris GitHub is notifying me of these comments just now... why? 😒 Sorry I missed it.
I have no ideas yet. The first step is for me to reproduce it.
jnm2
on 13 Jun 2017
@demonix I can't figure out a way to reproduce the error. Can you give your command line?
We have several people confirming that the dev build solved this problem for them, so the first thing that comes to mind is, are you absolutely sure teamcity was running the correct nunit console build?
jnm2
on 13 Jun 2017
I agree with @jnm2. You need to confirm that TC is running the dev build. It would be very easy for it to be using the copy of nunit3-console with which it is configured instead. Can you find something in the TC logs that say exactly what it is running?
CharliePoole
on 13 Jun 2017
Agreed, should make sure by double checking the log file. The same Team City log file which contains the exception stack trace at the end, would also clearly show the version number of Nunit Console at the beginning.
tfabris
on 13 Jun 2017
Hmmm... I sure hope our dev builds are showing the full version number with suffix! Might need to check that. IF it turns out it isn't, somebody should write an issue. I know hope it's available in the FileVersion.
CharliePoole
on 13 Jun 2017
Hi! There's no version info in log, but I'm 200% sure we're using correct build, as it is stored in our source repository and teamcity runs it from there.
Here'is contents of C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunner\ directory with file versions (as I understand only console runner was updated, please correct me, if I'm wrong)
Here's log from last failed build (June, 2) https://gist.github.com/demonix/000c49674255e0ba786f134e1f70450e
I've not tried to build this branch since June, 2, so I'm now trying to reproduce this again
And here's teamcity's builds list (to see, how often it fails)
demonix
on 13 Jun 2017
@demonix, you're right, the Team City logs don't show a version number for the Nunit output. I mis-remembered it. I was remembering how, on our system, we have the version number in the folder name where we store the Nunit console runner.
Your log:
Starting: C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunnernunit3-console.exe
Our log:
Starting: C:\BuildAgent\tools\NUnit.3.6.1nunit3-console.exe (we haven't upgraded yet)
That's why I was mis-remembering that the log would have the version number. :-)
In any case, you've shown that your console runner in that folder is 3.7.0
tfabris
on 13 Jun 2017
@demonix That's helpful, thanks. Can you provide your nunit console command line, replacing anything sensitive with stars?
jnm2
on 13 Jun 2017
@jnm2, gist with build log, that I've provided in previous comment contains command line. Is that what you asking for?
demonix
on 14 Jun 2017
@jnm2, this was the relevant part of the build log from @demonix:
[15:48:17] : [Step 4/8] Starting: C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunnernunit3-console.exe C:\buildAgent\temp\buildTmp\DCtcn8LCo3qtxPMzc0VKkaYYRQMl1Xwu.nunit --result=C:\buildAgent\temp\buildTmp\DCtcn8LCo3qtxPMzc0VKkaYYRQMl1Xwu.nunit.xml --noheader --framework=net-4.0 --explore=C:\buildAgent\temp\buildTmp\qhA40I9IGQVT8CghtajRnyQuaBOWsvrM_all.tests
[15:48:17] : [Step 4/8] in directory: C:\buildAgent\work\sources_all_unit_tests\Core\CI\UnitTesting\bin\Debug
tfabris
on 14 Jun 2017
@demonix @tfabris Oh yes, thanks.
jnm2
on 14 Jun 2017
@jnm2, I've just reproduced this when running from console.
I've took command line, generated by teamcity, and ran it multiple times manualy. A've got exception after 18 attempts. But after that I've ran it tens of times without getting exception :(
Btw, this issue reproduces not only with "--explore" option, but when actually performing tests:
[02:19:42] : [Step 4/8] Starting: C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunnernunit3-console.exe C:\buildAgent\temp\buildTmp\8HFFajQPDeLzyZvgiHHGcZcuqo9mflUP.nunit --result=C:\buildAgent\temp\buildTmp\8HFFajQPDeLzyZvgiHHGcZcuqo9mflUP.nunit.xml --noheader --framework=net-4.0 --testlist=C:\buildAgent\temp\buildTmp\vcfWeiRg9cDEggB8m5idXQ93Vb5UZ6Pf_step2.tests
[02:19:42] : [Step 4/8] in directory: C:\buildAgent\work\sources_all_unit_tests\Web\CI\Core\UnitTesting\bin\Debug
[02:19:42] : [Step 4/8] Runtime Environment
[02:19:42] : [Step 4/8] OS Version: Microsoft Windows NT 6.3.9600.0
[02:19:42] : [Step 4/8] CLR Version: 4.0.30319.42000
[02:19:42] : [Step 4/8]
[02:19:42] : [Step 4/8] Test Files
[02:19:42] : [Step 4/8] C:\buildAgent\temp\buildTmp\8HFFajQPDeLzyZvgiHHGcZcuqo9mflUP.nunit
[02:19:42] : [Step 4/8]
[02:19:42] : [Step 4/8] Test Filters
[02:19:42] : [Step 4/8] Test: Core.Web.UnitTesting.Tests.BillInfo.BusinessObjects.Factories.ServiceCenters.ServiceCenterInfoModelFactoryTest
[02:19:42] : [Step 4/8] Test: Core.Web.UnitTesting.Tests.BillInfo.BusinessObjects.Factories.Subscriptions.SubscriptionsInfoModelFactoryTest
[02:19:42] : [Step 4/8] Test: Core.Web.UnitTesting.Tests.BaseSite.Implementation.ExternalServicesUrlGeneratorTest
<...skipped...>
[02:20:09] : [Step 4/8] System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
[02:20:09] : [Step 4/8]
[02:20:09] : [Step 4/8] Server stack trace:
[02:20:09] : [Step 4/8] at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.Tcp.TcpFixedLengthReadingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[02:20:09] : [Step 4/8] at System.IO.BinaryReader.ReadBytes(Int32 count)
[02:20:09] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
[02:20:09] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
[02:20:09] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
[02:20:09] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryResponseMessage(Stream inputStream, IMethodCallMessage reqMsg, Boolean bStrictBinding)
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
[02:20:09] : [Step 4/8]
[02:20:09] : [Step 4/8] at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[02:20:09] : [Step 4/8] at NUnit.Engine.ITestAgent.Stop()
[02:20:09] : [Step 4/8] at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
[02:20:09] : [Step 4/8] at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
[02:20:09] : [Step 4/8] at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
[02:20:09] : [Step 4/8] at NUnit.Engine.Runners.MasterTestRunner.Dispose()
[02:20:09] : [Step 4/8] at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
[02:20:09] : [Step 4/8] at NUnit.ConsoleRunner.Program.Main(String[] args)
[02:20:09]W: [Step 4/8] Process exited with code -100
just for reference
MD5 of nunit3-console.exe is A8CA26C272A94532619B7D89A5A5C574
MD5 of nunit-agent.exe is D509CE40A004D000EB9632BD70A90C65
demonix
on 15 Jun 2017
@demonix All the insight you can provide is great. I've been stress testing the code thousands of times with various different command line parameters and haven't seen it blow up yet.
jnm2
on 15 Jun 2017
Can you tell me, does it blow up for you with just these CLI arguments? C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunner\nunit3-console.exe C:\buildAgent\temp\buildTmp\8HFFajQPDeLzyZvgiHHGcZcuqo9mflUP.nunit
jnm2
on 15 Jun 2017
The MD5s match what I'm running against (https://ci.appveyor.com/project/CharliePoole/nunit-console/build/3.7.0-dev-03644/artifacts)
Can you take as much as possible out of the command line? The biggest thing is if you can reproduce while specifying assemblies directly instead of passing it the .nunit file, and then see if you can repro with a single assembly, and then see how much you can take out of that assembly and still repro, till it's at the point where you're able to give us a clue how to reproduce it.
jnm2
on 15 Jun 2017
Actually, could you also share an anonymized version of your .nunit project file? That might provide insight into how many agents there are and what they're doing.
jnm2
on 15 Jun 2017
Hi!
Sorry for long delay, but I can't reproduce this in console yet :(
Teamcity generates .nunit per assembly, so it looks like this:
<NUnitProject>
<Settings activeconfig="active" appbase="C:\buildAgent\work\sources_all_unit_tests\Core\CI\UnitTesting\bin\Debug"/>
<Config name="active">
<assembly path="C:\buildAgent\work\sources_all_unit_tests\Core\CI\UnitTesting\bin\Debug\Core.UnitTesting.dll"/>
</Config>
</NUnitProject>
Is this equals to running nunit-console against single assembly, passed directly?
Btw, we also have different branches and some of them are still with v 3.6.1. Tests just crushed three times in a row in teamcity in such branch, but when I logged into build agent and tried to run same command, as teamcity did, everything worked well.
demonix
on 4 Jul 2017
Huh. I'm stumped. We may have to wait until 3.7 is released and twist your arm to move everything to that.
jnm2
on 8 Jul 2017
@demonix You're not the only person who was still getting SocketExceptions. I went over the .NET Framework source code again and discovered a flaw in #223. Armed with new information, I think this is likely our fix: https://github.com/nunit/nunit-console/commits/socketexception_2
Would you very kindly let us know the outcome of running your tests against https://ci.appveyor.com/api/buildjobs/ovhbgefoo1m92jnj/artifacts/package%2FNUnit.ConsoleRunner.3.7.0-ci-03745-socketexcep.nupkg a couple times? Thanks!
jnm2
on 14 Jul 2017
Hi!
We've updated console-runner and nunit-agent to the suggested version and it crashed.
Starting: C:\buildAgent\work\sources_all_unit_tests\Deploy\ConsoleTestRunner\nunit3-console.exe C:\buildAgent\temp\buildTmp\JhirONxKAOuJo4uamAXrYQIHCjX8mJAF.nunit --result=C:\buildAgent\temp\buildTmp\JhirONxKAOuJo4uamAXrYQIHCjX8mJAF.nunit.xml --noheader --framework=net-4.0
<skipped>
[09:53:39] : [Step 4/8] System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
[09:53:39] : [Step 4/8]
[09:53:39] : [Step 4/8] Server stack trace:
[09:53:39] : [Step 4/8] at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.Tcp.TcpFixedLengthReadingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
[09:53:39] : [Step 4/8] at System.IO.BinaryReader.ReadBytes(Int32 count)
[09:53:39] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
[09:53:39] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
[09:53:39] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
[09:53:39] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
[09:53:39] : [Step 4/8] at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryResponseMessage(Stream inputStream, IMethodCallMessage reqMsg, Boolean bStrictBinding)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
[09:53:39] : [Step 4/8]
[09:53:39] : [Step 4/8] Exception rethrown at [0]:
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
[09:53:39] : [Step 4/8] at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
[09:53:39] : [Step 4/8] at NUnit.Engine.ITestAgent.Stop()
[09:53:39] : [Step 4/8] at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
[09:53:39] : [Step 4/8] at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
[09:53:39] : [Step 4/8] at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
[09:53:39] : [Step 4/8] at NUnit.Engine.Runners.MasterTestRunner.Dispose()
[09:53:39] : [Step 4/8] at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
[09:53:39] : [Step 4/8] at NUnit.ConsoleRunner.Program.Main(String[] args)
[09:53:39]W: [Step 4/8] Process exited with code -100
I also discovered, that teamcity adds "--teamcity" parameter (at least) and than reads output. Maybe this somehow slows down, or vice versa, speeds up console and triggers this exception.
demonix
on 14 Jul 2017
Thanks @demonix.
jnm2
on 14 Jul 2017
I just got the same problem in a set of tests that execute just fine in ReSharper's test runner. Not having the stack trace of the failing test available makes this very hard to debug. This is all I have to work with:
```c#
An existing connection was forcibly closed by the remote host
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders request
Headers, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
```
I'm using nunit3-console.exe version 3.7.0, running locally, and I'm only passing in the path to three .dll files, not using --explore or anything else. The test execution also fails in TeamCity.
asbjornu
on 20 Sep 2017
Interesting. Adding the --inprocess argument made nunit3-console.exe process crash, so by building nunit-console locally and copying the build output into the problematic project, I was able to attach to the nunit3-console.exe process and figure out that there was some weird edge case code being triggered that caused a StackOverflowException. Fixing the problematic code also fixed the test execution. 😄 👍
It would be nice to know why the test execution failed, though. Just the idea of it being a StackOverflowException would have been very helpful. Even better if a source file and line number could be pinpointed, somehow.
asbjornu
on 20 Sep 2017
In the interest of full disclosure, my intent is to not spend more time on patching the current remoting system but to replace it with a lighter weight manual implementation. Right now it's second from the top of my todo list. However, if you spend time diagnosing the situation and believe you have a fix, I'm happy to help get that in to help people short-term.
jnm2
on 21 Sep 2017
@jnm2 So what's the status of this issue then? It shows as Open but with a label of closed:done and assigned to you.
CharliePoole
on 21 Sep 2017
@CharliePoole Yes, that is confusing. What should I tag it as, wontfix? It's not a duplicate of https://github.com/nunit/nunit-console/issues/266 but it is superseded by it.
jnm2
on 21 Sep 2017
@jnm2 Well, I was thinking somebody else might fix it since you said you didn't want to work on it any more. I don't think closing it as won't fix is right unless that's Rob's decision. It's a reported bug that apparently still exists.
If you merely no longer want to work on it personally, then I'd say to just take your name off it.
CharliePoole
on 21 Sep 2017
Works for me. It makes sense to leave this issue open until remoting is replaced in case anyone comes up with a suggestion to fix the remoting issue.
jnm2
on 21 Sep 2017
@asbjornu unfortunately, StackOverflowException is one of the exceptions that cannot be caught, because of that, the agent just crashes and doesn't get a chance to report back to the console runner why it crashed. There are several other exceptions like this.
I have been suspicious that many of these errors that users report are actually uncaught exceptions in their test code and have nothing to do with NUnit. It could be that this issue is actually fixed in NUnit.
One possible solution would be to catch the SocketException in the console and report a more friendly error message like "The test agent has crashed. This is often because the code being tested threw an exception that cannot be caught and crashed the agent. To diagnose the problem, we recommend running your tests with the --inprocess command line option".
rprouse
on 22 Sep 2017
@rprouse It's probably a matter of personal preference, but I use the --debug option to debug the agent. It requires an extra step by the user but it runs the tests in the same environment (under the agent), which is an advantage. Both are, of course, available to users.
CharliePoole
on 22 Sep 2017
@rprouse: Yes, a wrap around the SocketException providing that message would be very helpful.
asbjornu
on 22 Sep 2017
@rprouse That's a really good angle!
jnm2
on 22 Sep 2017
Just my two cents, but please don't close the issue. We have the same issue on TeamCity. I've been watching this issue hoping for some resolution. We've tried upgrading to 3.7.0 but that didn't cause anything to change. I doubt that it is something in our code, otherwise I would expect it to surface locally running through the normal VS nunit adapter. It's been happening for months now. Roughly 1 out of every three runs will fail with this exception. I don't want to sound unappreciative though. I really appreciate all the work you guys have done, and have put into such a hard to track down problem.
sethflowers
on 22 Sep 2017
@rprouse - You said "I have been suspicious that many of these errors that users report are actually uncaught exceptions in their test code".
This idea is inconsistent with the behavior that I have seen related to the socket exception bug, which is:
- The bug did not occur in an earlier version of Nunit.
- The bug only started occurring when we upgraded to a new version of Nunit.
- The bug occurs randomly, it does not trigger on the same test cases.
- The bug is rare.
- The bug can occur during test case runs which do not access sockets.
Catching the socket exception and throwing a more friendly error message would merely cloud the issue.
tfabris
on 22 Sep 2017
The bug only started occurring when we upgraded to a new version of Nunit.
That means, in theory, we could do a git bisect to identify the commit that introduces the issue. That might shed light on it.
jnm2
on 22 Sep 2017
I see hope... I'm interested again. 😄
jnm2
on 22 Sep 2017
@jnm2 - Our version jump, between the time that we didn't have the bug, and the time that we started encountering the bug, was very large. We jumped from 2.6.4 to 3.6.1. So I don't know if there is an easy way to locate the version which induced the bug. Maybe someone else has a narrower range.
tfabris
on 22 Sep 2017
Title of this issue says we are talking about an error with the --explore option. Some folks have commented that they have the "same issue" but are not using --explore. As a start to narrowing it down - or deciding if we actually have two issues - it would be useful if those participating in this thread clarify whether they are talking about the "explore issue" or something more general. Using --explore versus running tests causes us to execute different code, although some of it overlaps.
Exception bugs are just like that. The "same bug" is seen by many people for different reasons in the code. 😞
CharliePoole
on 22 Sep 2017
My answer to the --explore option: We only encounter the issue when running our test cases under Team City using Team City agents. I believe this means that the --explore option is always used, since I think that is part of the way Team City sets up and runs the tests.
tfabris
on 27 Sep 2017
@tflabris I didn't realize that about TeamCity. That explains a lot!
CharliePoole
on 27 Sep 2017
Team city has a feature where it re-orders the test cases, so that tests which failed on a prior buildrun are first in the execution order of the next buildrun. This is so that you can check in fixes and then see immediately whether or not your fixes pass the tests.
I think (not certain) that's what it's using the --explore option for. I think it does that every time, regardless of whether or not it needs to reorder the test cases.
tfabris
on 27 Sep 2017
Hello, we have recently started seeing this error following a rebuild of our [TeamCity] build agents where we moved from (zip versions of) nunit-console 3.2.1 to 3.7.0.
We've only seen it a few times in the last ~25-30 builds, but so far it always fails during the unit tests (rather than our integration tests - i.e. no networking/sockets). Our code is split across multiple projects, so there are multiple test projects which run sequentially. The error can seemingly happen during any one of them (sometimes the first, sometimes the 10th project).
The referenced version of NUnit within the projects hasn't changed (3.7.1), only the console runner (from 3.2.1 to 3.7.0) - _might_ help with the suggestion above of narrowing down where the issue was introduced? I will try and find some time to set up an environment to help with this.
Sample stack trace from TeamCity:
Starting: C:\BuildTools\NUnit\nunit3-console.exe C:\BuildAgent\temp\buildTmp\AQw9t6DtUajuQYzkBDyELUl2u0SWUu6l.nunit --result=C:\BuildAgent\temp\buildTmp\AQw9t6DtUajuQYzkBDyELUl2u0SWUu6l.nunit.xml --noheader --framework=net-4.0 --teamcity --explore=C:\BuildAgent\temp\buildTmp\WgxUVcsRrdmHsYWFTRiZSkkrjYIHsqbg_all.tests
in directory: C:\BuildAgent\work\175a26dc95f5b9af\src\Agents\Agent1.Tests\bin\Release
Runtime Environment
OS Version: Microsoft Windows NT 10.0.15063.0
CLR Version: 4.0.30319.42000
Test Files
C:\BuildAgent\temp\buildTmp\AQw9t6DtUajuQYzkBDyELUl2u0SWUu6l.nunit
System.Net.Sockets.SocketException (0x80004005): An existing connection was forcibly closed by the remote host
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.Tcp.TcpFixedLengthReadingStream.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.IO.BinaryReader.ReadBytes(Int32 count)
at System.Runtime.Serialization.Formatters.Binary.SerializationHeaderRecord.Read(__BinaryParser input)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadSerializationHeaderRecord()
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryResponseMessage(Stream inputStream, IMethodCallMessage reqMsg, Boolean bStrictBinding)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestAgent.Stop()
at NUnit.Engine.Runners.ProcessRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.MasterTestRunner.Dispose()
at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter)
at NUnit.ConsoleRunner.Program.Main(String[] args)
Process exited with code -100
Jevonius
on 10 Oct 2017
A minor comment: your command-line includes both --explore and --result. The first causes the action to be explore and saves the result. The second says where to save the execution result but no execution takes place because explore was specified. This probably isn't the cause of the problem, but it's a little odd.
CharliePoole
on 10 Oct 2017
Hi Charlie, I agree very odd - I've just had a look at ConsoleRunner.cs and can see the switch you mean. Perhaps the TeamCity logging is lying, and it runs the console twice (first to explore, then to run) and logs it oddly?
I've been trying to replicate the bug outside of TeamCity but am having no luck - it never fails! At this point, I'm considering making a debug build from 3.7.0's source, perhaps with an addition to public static int Main(string[] args) which just spits out the raw args value(s) on each execution (to try and see if the TC logging is lying). Bit pushed for time so I haven't got time to spelunk fully into the nunit3-console source, but does that sound worthwhile? I'll then put the debug build on our build agents and hopefully catch another failure with a bit more detail in the stack trace!
Potentially notable, we have yet to see a failure in our Integration Tests. These are run with the parameters --workers=1 --agents=1 as otherwise our database tests get confused (they're not written with parallel execution in mind).
Jevonius
on 10 Oct 2017
We have had a few more failures now, at different points - TeamCity _does_ run the console twice for each test DLL - once with explore for each DLL, and then again with testlist for each. At a guess, this might relate to the re-ordering @tfabris covers above (and/or for some other reporting that TC does).
I suspect the result is included for both automatically. Example calls:
Starting: C:\BuildTools\NUnit\nunit3-console.exe C:\BuildAgent\temp\buildTmp\emFgXb0jqipVxiXimC9W8kX2sdvbUpi9.nunit --result=C:\BuildAgent\temp\buildTmp\emFgXb0jqipVxiXimC9W8kX2sdvbUpi9.nunit.xml --noheader --framework=net-4.0 --teamcity --explore=C:\BuildAgent\temp\buildTmp\aCCxdLrsDMRhfxC5P1LzSwAUGHPm7Gfa_all.tests
and
Starting: C:\BuildTools\NUnit\nunit3-console.exe C:\BuildAgent\temp\buildTmp\emFgXb0jqipVxiXimC9W8kX2sdvbUpi9.nunit --result=C:\BuildAgent\temp\buildTmp\emFgXb0jqipVxiXimC9W8kX2sdvbUpi9.nunit.xml --noheader --framework=net-4.0 --teamcity --testlist=C:\BuildAgent\temp\buildTmp\DcpadqF2dKgUFl2d4MaLFiDFMpVZt40k_step2.tests
There are examples (in different builds) where both variants fail. The stack trace is identical, except for (as expected) the second to last line which is either:
at NUnit.ConsoleRunner.ConsoleRunner.ExploreTests(TestPackage package, TestFilter filter)
or
at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
It's weird and confusing to users but not to NUnit. The --result is ignored when exploring. Probably, the TC devs decided it was simpler to just include that parameter all the time.
CharliePoole
on 10 Oct 2017
This has all of a sudden started happening for us too 😢 - NUnit Console 3.6.1
kdubau
on 30 Oct 2017
@kdubau 'sudden' sounds like it has a cause. That's good news! 😄
Can you add --trace=Debug, and see if you get anything useful in the agent logs? We've been seeing this before as a communication protocol crash - but jnm2 rightly brought up that we see the same stack trace whenever the agent process crashes. 😞
ChrisMaddock
on 30 Oct 2017
"This has all of a sudden started happening for us too 😢 - NUnit Console 3.6.1"
And in case its helpful, I found my way here as I just encountered this for the first time today as well. Our scenario matches most others described - TeamCity, recently converted to nunit3, running 3.6.1
I've had 2 consecutive build failures today which fail on different test DLLs, and the check in that triggered the first failed build had no changes that would affect the failing tests.
I've just setup a run with the inprocess option to see what effect that has, and will then try 3.7
garyjhill
on 30 Oct 2017
My two cents. We've had the same problem, turns out it was an unhandled exception in the code (failed malloc when calling C++ function). It's probably worth capturing a dump of the process and running it under VS or windbg. (also, in our case happened both on 3.4.1 and 3.7.0, but as I said the problem was not on NUnit)
jpmnteiro
on 30 Oct 2017
Normally I'm the first to caution against blaming compiler/library/OS/etc for problems. In this case however no code under test has changed since we had consistent passes.
The upgrade to 3.7 appears to have solved the issue for us.
garyjhill
on 31 Oct 2017
@ChrisMaddock you would never believe that when I added --trace=Debug it started working...
Going to try a few more times to get a repro with the trace enabled.
kdubau
on 1 Nov 2017
Maybe the logging's enough to slow it down and the race condition to become less likely. 😞
@jnm2's working on a more long-term fix at https://github.com/nunit/nunit-console/issues/266 - which will hopefully help.
ChrisMaddock
on 1 Nov 2017
Yea, I was about to suggest this smells much like a race condition. I've yet been able to reproduce the failure with --trace=Debug enabled, and when leaving it absent it fails every time. I'll use this as a work around for now.
The only difference I can really think about in this assembly which is erroring is that it's far more dependent on testcasesources than some of our other assemblies.
Our assembly is also heavily dependent on TestCaseSource. Perhaps the reason it started "all of a sudden" for us is a growing set of TestCaseSources?
kdubau
on 1 Nov 2017
@kdubau - Sorry, I didn't clock the version from your first post. NUnit Console 3.7 had some significant improvements here. I'd try upgrading - that seems to have sorted the vast majority of cases. (Also work by @jnm2 ✨)
ChrisMaddock
on 1 Nov 2017
Thanks @kdubau. I finally decided to just try adding --trace=Debug as a parameter, and I was able to run the tests 10 times in a row without fail.
Prior to this, our runs fail roughly half the time. This has been our teams sad state of affairs for months now. We've gotten to the point of often just disabling unit tests to get a QA build out, having to run the tests locally. We are on 3.7.0, so whatever fixes were supposed to be in that version haven't helped us.
Our facts:
- Solution has 108 projects, almost half of which are test projects, containing roughly 4900 tests.
- We are on TeamCity Professional 2017.1.4 (build 47070)
- NUnit runner - NUnit 3
- Path to NUnit Console: packages\NUnit.ConsoleRunner.3.7.0\toolsnunit3-console.exe
- Additional command line parameters - --skipnontestassemblies --trace=Debug
Note that this situation seems to have gotten worse with the more test projects we have. I've thought about ILMerging all the test dlls into a single library and running nunit against that one library, but that opens up another can of worms.
sethflowers
on 20 Dec 2017
Hey, our team also sees the same when using NUnit Console 3.7.0 on TeamCity. We've added --trace=Debug as per the above recommendations which has resulted in a drop of the exception appearing. However, it still happens every now and then, so adding the above parameter _does not_ fully fix this issue.
asendimitrov
on 21 Dec 2017
The trace may slow things down enough to reduce the problem incidence, but it's real purpose is to collect information. Have you examined the trace from a failing incident?
CharliePoole
on 21 Dec 2017
We're seeing intermittent failures like that fairly often in our CI builds, creating distracting volatility and reducing trust in our (quite sizeable) test suite.
Is there any extra checks/protection/heartbeats/dumps agent process can do to ensure if it ever crashes abruptly the amount of diagnostic is relevant to hint at the cause of failure?
Can agent listen to unhandled exception event and capture/pass to console all the stacks of all threads, clearly marking NUnit-specific threads, thread that crashed and all the rest?
Additionally, if agent crashes executing test N, can the host restart the agent and keep on starting from N+1? All the crash info, stacks and details can be attached to the test during which it all happened.
Those two features would give pretty thorough production-level diagnostics at the right time and right place.
Many thanks!
mihailik
on 21 Dec 2017
Hi @CharliePoole, thanks for your quick response.
I may have not used the best words there, I don't really expect this to actually "fix" the issue, was just highlighting the workaround doesn't work from every attempt as a response to other people above saying they can't reproduce with the parameter in place.
Looking at the internal trace log, the exception in there is what was already posted above in this comment thread:
12:30:12.705 Error [ 1] ProcessRunner: Failed to run remote tests An existing connection was forcibly closed by the remote host
12:30:12.709 Info [ 1] ProcessRunner: tests.csproj
12:30:14.714 Warning [ 1] ProcessRunner: Failed to unload the remote runner. No connection could be made because the target machine actively refused it 127.0.0.1:63249
12:30:14.714 Error [ 1] ProcessRunner: No connection could be made because the target machine actively refused it 127.0.0.1:63249
12:30:14.714 Debug [ 1] ProcessRunner: Stopping remote agent
12:30:16.717 Error [ 1] ProcessRunner: Failed to stop the remote agent. No connection could be made because the target machine actively refused it 127.0.0.1:63249
This has occurred 13 minutes after the tests build has started running and has resulted in a preemptive failure of that build. By the way, our tests use the [assembly: Parallelizable(ParallelScope.Fixtures)] attribute if that makes any difference.
asendimitrov
on 21 Dec 2017
That's the trace for the console runner and engine. It has the info that the agent failed, which we already knew. You should have two other trace files, one for the agent and one for the framework. We want to figure out __why__ the agent exited and I'm hoping one of those will tell us.
CharliePoole
on 21 Dec 2017
Apologies for the misunderstanding. I'm posting the data from all the log files that exist after a failed run (one of these is again the console runner and engine trace but that's just so you can compare the times).
---- nunit-agent_41052 ----
...
08:16:06.635 Info [ 5] DomainManager: Creating AppDomain domain-23d3bc9a-Tests.csproj
08:16:06.638 Debug [ 5] ServiceManager: Request for service IDriverService satisfied by DriverService
08:16:06.673 Info [ 5] NUnitFrameworkDriver: Loading Tests.dll - see separate log file
08:16:07.031 Info [ 5] NUnitFrameworkDriver: Loaded Tests.dll
08:16:07.031 Info [ 5] NUnitFrameworkDriver: Running Tests.dll - see separate log file
<end of file>
---- InternalTrace.41052.Tests.dll -----
...
08:23:30.229 Debug [15] Dispatcher: Using Parallel strategy for Feature1 OneTimeTearDown
08:23:30.229 Info [15] TestWorker: Worker#7 executing Feature1 OneTimeTearDown
08:23:30.230 Info [15] TestWorker: Worker#7 executing Feature2
08:23:30.230 Debug [15] Dispatcher: Using Direct strategy for Scenario1
08:23:30.371 Debug [11] Dispatcher: Using Direct strategy for Scenario2
<end of file>
---- InternalTrace.38552 ----
...
08:16:06.208 Debug [ 1] TestAgency: Launched Agent process 41052 - see nunit-agent_41052.log
08:16:06.208 Debug [ 1] TestAgency: Command line: "F:\TCAgent\work\53f0269d1fc0fabc\src\packages\NUnit.ConsoleRunner.3.7.0\tools\nunit-agent.exe" b6859202-966d-4a07-bc99-fee046ec0808 tcp://127.0.0.1:65096/TestAgency --pid=38552 --trace:Debug
08:16:06.208 Debug [ 1] TestAgency: Waiting for agent {b6859202-966d-4a07-bc99-fee046ec0808} to register
08:16:06.610 Debug [ 1] TestAgency: Returning new agent {b6859202-966d-4a07-bc99-fee046ec0808}
08:23:30.429 Error [ 1] ProcessRunner: Failed to run remote tests An existing connection was forcibly closed by the remote host
08:23:30.433 Info [ 1] ProcessRunner: Unloading Tests.csproj
08:23:32.435 Warning [ 1] ProcessRunner: Failed to unload the remote runner. No connection could be made because the target machine actively refused it 127.0.0.1:65097
08:23:32.435 Error [ 1] ProcessRunner: No connection could be made because the target machine actively refused it 127.0.0.1:65097
08:23:32.435 Debug [ 1] ProcessRunner: Stopping remote agent
08:23:34.431 Error [ 1] ProcessRunner: Failed to stop the remote agent. No connection could be made because the target machine actively refused it 127.0.0.1:65097
...
That points out a small bug in ProcessRunner. It gets an error because the connection was forcibly closed by the remote runner, that is, the agent has terminated. But it still tries to unload the remote runner and (naturally) gets another error. We should flag the runner as terminated and not try to unload it.
That wouldn't help you, of course. We still need to get more info about what caused the initial error.
CharliePoole
on 2 Jan 2018
Yeah, unfortunately in my case the other trace files don't have any information on what has happened. The logging seems to just stop right before the socket exception occurs in the console trace. I'll keep watching this closely and if by any chance something additional pops up I'll let you guys know. In the meantime I really hope that something useful appears on someone else's screen as this really causes a headache.
asendimitrov
on 3 Jan 2018
Seem to have run into the same issue, however the below stack trace occurred immediately after a "Process is terminated due to StackOverflowException" message. Hopefully this information helps.
System.Net.Sockets.SocketException (0x80004005): No connection could be made because the target machine actively refused it 127.0.0.1:13313
Server stack trace:
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket(EndPoint ipEndPoint)
at System.Runtime.Remoting.Channels.SocketCache.GetSocket(String machinePortAndSid, Boolean openNew)
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestS
tream)
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream,
ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Unload()
at NUnit.Engine.Runners.ProcessRunner.UnloadPackage()
at NUnit.Engine.Runners.AbstractTestRunner.Unload()
at NUnit.Engine.Runners.MasterTestRunner.UnloadPackage()
at NUnit.Engine.Runners.AbstractTestRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
at NUnit.ConsoleRunner.Program.Main(String[] args)
jordan-mace
on 17 Jan 2018
Since our team got sad too about the failing builds I've patched the console runner. The workaround does not solve the problem. It deals with the consequences of the problem in a way that allows our team being able to work with nunit effectively.
https://github.com/eberlid/nunit-console/tree/issue-171
Again, this does not solve the problem. Thanks for everyone still investigating in solving the real problem behind that.
eberlid
on 17 Jan 2018
@jordz0005 - we can't handle a StackOverflowException - but shouldn't crash the entire console, if you're not running inprocess. Will you try running against the latest build of master and see if https://github.com/nunit/nunit-console/pull/346 fixes your problem? You can get this from the MyGet feed - there's a link in the readme.
@eberlid - I can't find anywhere you've every shared your exact stack trace - would you be able to, plus the debug logs from a crashing build? There's a range of different errors causing these socketexceptions. We've fixed several in master - you may also wish to try the MyGet build, and see if any fix your particular variation of the issue. (Although it may be worth waiting for #177 to be merged first, which I believe may also occasionally cause SocketExceptions.)
ChrisMaddock
on 18 Jan 2018
All,
We believe this issue will have been fixed by @BlythMeister's temporary fix in #370.
Until the next console release, please try the master build, version 3.9.0-dev-03932 or later. You can find this on out MyGet Feed.
If you still experience this issue with the above build of master...
Run your tests with the
--inprocessflag - see if this exposes a more useful exception or error message.Add the
--trace=Debugflag to your command line. When you next see the exception, examine the logs for any clues.
Please then open a new issue. The SocketException is often caused by an nunit-agent crash, which can occur in a number of different scenarios. We believe many of these have been fixed in NUnit.Console 3.8, and the fix in #372 - however, they may still be edge cases we have yet to tackle. Please report these if you come across any!
ChrisMaddock
on 15 Feb 2018
I am still getting the same issue in Recent version of nunit console and nunit framework. Below are the details:
NUnit Console Runner 3.8.0
Copyright (c) 2018 Charlie Poole, Rob Prouse
Runtime Environment
OS Version: Microsoft Windows NT 10.0.10586.0
CLR Version: 4.0.30319.42000
Test Files
D:\SomePath\MyFile.dll
Unhandled Exception: NUnit.Engine.NUnitEngineException: Remote test agent exited with non-zero exit code -1073741571
at NUnit.Engine.Services.TestAgency.OnAgentExit(Object sender, EventArgs e)
at System.Diagnostics.Process.OnExited()
at System.Diagnostics.Process.RaiseOnExited()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object
state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, B
oolean preserveSyncCtx)
at System.Threading._ThreadPoolWaitOrTimerCallback.PerformWaitOrTimerCallback(Object state, Boolean timedOut)
Errors, Failures and Warnings
1) Error :
System.Net.Sockets.SocketException : An existing connection was forcibly closed by the remote host
--SocketException
Server stack trace:
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Runtime.Remoting.Channels.SocketStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Runtime.Remoting.Channels.SocketHandler.ReadFromSocket(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.Read(Byte[] buffer, Int32 offset, Int32 count)
at System.Runtime.Remoting.Channels.SocketHandler.ReadAndMatchFourBytes(Byte[] buffer)
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadAndMatchPreamble()
at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadVersionAndOperation(UInt16& operation)
at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders request
Headers, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at NUnit.Engine.ITestEngineRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter)
Test Run Summary
Overall result: Failed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
Start time: 2018-03-23 12:10:33Z
End time: 2018-03-23 12:10:40Z
Duration: 6.897 seconds
ShaVikas
on 23 Mar 2018
@ShaVikas We'd be grateful if you'd be able to let us know if the most recent changes resolve the issue you're having before we release them. Can you merge this with your nuget.config and then use version 3.9.0-dev-03932 or later, as @ChrisMaddock mentions above?
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<packageSources>
<add key="NUnit prerelease" value="https://www.myget.org/F/nunit/api/v3/index.json" />
</packageSources>
</configuration>
@jnm2's suggestion is good.
However - I think that exit code you have is caused by a stackoverflow exception in the NUnit Agent.
Of course, NUnit can't do a lot to catch a stackoverflow exception - if that is the case, you'd need to debug and work out what in your tests is causing it. (First step, I'd run with the --inprocess flag, to verify that it is a stack overflow you're seeing - if the stack overflow occurs in the main process, you should see a nicer error message.)
One day, it would be good to recognise the agent exit code and add nicer debug handling.
ChrisMaddock
on 23 Mar 2018
Since the original issue was about an exception thrown using --explore only and not when running tests, this new report is not the "same issue."
CharliePoole
on 23 Mar 2018
@ChrisMaddock Thanks, with the --inprocess flag, it confirms stack overflow.
ShaVikas
on 23 Mar 2018
For socket exceptions, we could print out a suggestion to use --inprocess to diagnose the issue.
jnm2
on 23 Mar 2018
@ShaVikas Great!
I wrote up my thoughts on this in https://github.com/nunit/nunit-console/issues/391. If you'd be interested in submitting a pull request, it should be something we can improve fairly easily. 😄
ChrisMaddock
on 23 Mar 2018
Related issues
elinato
·
7Comments
lbergnehr
·
3Comments
ChrisMaddock
·
3Comments
CharliePoole
·
8Comments
CharliePoole
·
6Comments
Most helpful comment
@asbjornu unfortunately,
StackOverflowExceptionis one of the exceptions that cannot be caught, because of that, the agent just crashes and doesn't get a chance to report back to the console runner why it crashed. There are several other exceptions like this.I have been suspicious that many of these errors that users report are actually uncaught exceptions in their test code and have nothing to do with NUnit. It could be that this issue is actually fixed in NUnit.
One possible solution would be to catch the
SocketExceptionin the console and report a more friendly error message like "The test agent has crashed. This is often because the code being tested threw an exception that cannot be caught and crashed the agent. To diagnose the problem, we recommend running your tests with the --inprocess command line option".