Eventstore: StackOverflowException when subscribing to all events

[urbanhusky]

Issue

After processing most or all events (from SubscribeToAllFrom), I end up with a StackOverflowException

An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

Replication of the issue

I've replicated an issue I'm having with the following code

Event store facade

``` C#
namespace EventStoreStackOverflowTest
{
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using global::EventStore.ClientAPI;
using global::EventStore.ClientAPI.SystemData;

public class EventStore
{
    private IEventStoreConnection connection = null;

    public void SubscribeToAllEvents_TestOverflow()
    {
        if (connection == null)
            this.Connect();

        this.connection.SubscribeToAllFrom(
            Position.Start,
            new CatchUpSubscriptionSettings(100, 100, true, true),
            (sub, evt) =>
            {
                if (evt.Event.EventNumber % 100000 == 0)
                {
                    Debug.WriteLine(evt.Event.EventNumber);
                }
            },
            userCredentials: new UserCredentials("admin", "changeit"));
    }

    private void Connect()
    {
        this.connection = EventStoreConnection.Create(new IPEndPoint(IPAddress.Loopback, 1114));
        this.connection.ConnectAsync().Wait();
    }
}

}

This is being started from a console application (originally I had it in a WPF application but it seems to be independent of the application type):
## Console application code

``` C#
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace EventStoreStackOverflowTest
{
    class Program
    {
        static void Main(string[] args)
        {
            var eventStore = new EventStore();
            eventStore.SubscribeToAllEvents_TestOverflow();
            Console.WriteLine("Done");
            Console.ReadLine();
        }
    }
}

My instance of Event Store is listening on TCP port 1114 and has ~7.5 million events.
I'm using the latest NuGet package (3.6.2)

[urbanhusky]

P.S.: This happens with a read batch size below ~650.

[gregoryyoung]

Where is the StackOverflowException occuring?

On Fri, May 20, 2016 at 11:45 AM, urbanhusky [email protected]
wrote:

P.S.: This happens with a read batch size below ~650.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/923#issuecomment-220549967

Studying for the Turing test

[dnauck]

I had the same issue with the default settings on a CatchUpSubscription to $all with Client 3.6

The complete IIS process crashed without stacktrace & co.

[urbanhusky]

The StackOverflowException occurs somewhere deep within mscorlib.dll, I don't have the symbols loaded for that.

System.StackOverflowException was unhandled
Message: An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll

The entire call stack looks like this:

    mscorlib.dll!System.Threading.Tasks.Task.AddExceptionsFromChildren()    Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageThree() Unknown
    mscorlib.dll!System.Threading.Tasks.Task.FinishStageTwo()   Unknown
    mscorlib.dll!System.Threading.Tasks.Task.ProcessChildCompletion(System.Threading.Tasks.Task childTask)  Unknown
...

And ends with

The maximum number of stack frames supported by Visual Studio has been exceeded.    

[urbanhusky]

To clarify:
new CatchUpSubscriptionSettings(100, 100, true, true) is a problem
new CatchUpSubscriptionSettings(100, 650, true, true) seems to be around the lower bound where this doesn't occur with my sample set.

[dnauck]

Until @urbanhusky posted this issue i was not even sure what caused the stack overflow exception. But now it looks like we have the same issue.

How the problem occurs:

• a CatchupSubscription to $all with default queue/batch size settings, resolve linktos: true
• starting from the beginning
• EventAppeared event handler was only checking for events of a special type and returned immediately (as to this time no event of this type was written)
• near the point or at the point of switching from history mode to live mode the application crashed

Some more details on my environment:

Configuration:

• IIS web application
• EventStore HA 3.6.0 (without projections, default settings) - 4.5 mio events
• EventStoreClient 3.6.0
• .NET 4.5.1 / Win2k12 R2 (German), 4 CPU, 16 GB RAM

I'm not allowed to install any other software on this environment so i cannot debug this iis process crash with external tools.

Error message in windows event log:

Name der fehlerhaften Anwendung: w3wp.exe, Version: 8.5.9600.16384, Zeitstempel: 0x5215df96
Name des fehlerhaften Moduls: mscorlib.ni.dll, Version: 4.0.30319.34209, Zeitstempel: 0x53489fcf
Ausnahmecode: 0xc00000fd
Fehleroffset: 0x000000000051567c
ID des fehlerhaften Prozesses: 0xb1c
Startzeit der fehlerhaften Anwendung: 0x01d1b0e37b23c112
Pfad der fehlerhaften Anwendung: c:\windows\system32\inetsrv\w3wp.exe
Pfad des fehlerhaften Moduls: C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\701f2b79b02a02beba70e50bb2edb212\mscorlib.ni.dll
Berichtskennung: 35d605fd-1cd7-11e6-8184-005056b30f73
Vollständiger Name des fehlerhaften Pakets: 
Anwendungs-ID, die relativ zum fehlerhaften Paket ist: 

I resolved this issue by:

• disabling this subscription
• added a new event of the event type the subscription is listening for (then the EventAppeared event handler persist the checkpint/position and starts the subscription after the crash from this new position without crashing again)

I was not abel to reproduce it on:

• my laptop with Win 8.1
• on Linux (SLES11SP2) + Mono 4.x

[pgermishuys]

This is a bug introduced in the 3.6.0 client by this commit https://github.com/EventStore/EventStore/pull/837/commits/60455a0c82e8085a2838c4f8731a8804a0ed4652 it would seem.

[pgermishuys]

There will be a 3.6.3 client out on NuGet today without that commit that should solve this issue. We will look into working on that PR and fixing this issue in 3.6.4 or in 3.7.0

[hayley-jean]

The 3.6.3 client binaries are now on NuGet. Please let us know if you have any issues

[pgermishuys]

@urbanhusky Let us know if the problem was resolved in 3.6.3 so that we can close this issue. Thank you!

[urbanhusky]

@pgermishuys Give me a minute please, I just arrived in the office :)
I can confirm that 3.6.3 no longer causes this exception. The Process Memory also appears to be much more stable.

Thanks for the quick fix.

[dnauck]

I can also confirm that Client 3.6.3 fixed this issue on my system. Thanks for the fast solution.