Signalr: MapSignalR hangs in the creation of the Performance Counters

Created on 23 Feb 2015 · 33Comments · Source: SignalR/SignalR

Hi,
With the 2.2.0 build I started to have a problem in the app.MapSignalR() which it gets stuck in creating the Performance Counters. It's hard to reproduce but I get it about 3-4 times a day.

This is the callstack:

    System.dll!System.Diagnostics.PerformanceCounterLib.GetPerformanceCounterLib(string machineName, System.Globalization.CultureInfo culture)  Unknown
    System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown
    System.dll!System.Diagnostics.PerformanceCounter.InitializeImpl()   Unknown
>   System.dll!System.Diagnostics.PerformanceCounter.PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)  Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()  Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(string instanceName, System.Threading.CancellationToken hostShutdownToken)   Unknown
    Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver resolver, string instanceName, System.Threading.CancellationToken hostShutdownToken)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.UseSignalRMiddleware<Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware>(Owin.IAppBuilder builder, object[] args)   Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR.AnonymousMethod__0(Owin.IAppBuilder subApp)    Unknown
    Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, Microsoft.Owin.PathString pathMatch, System.Action<Owin.IAppBuilder> configuration) Unknown
    Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, string pathMatch, System.Action<Owin.IAppBuilder> configuration)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, string path, Microsoft.AspNet.SignalR.HubConfiguration configuration)    Unknown
    Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder)  Unknown
    SnowSparkle.OneSaas.dll!SnowSparkle.OneSaas.SignalRConfig.Configuration(Owin.IAppBuilder app) Line 22   C#
    [Native to Managed Transition]  
    Microsoft.Owin.Host.SystemWeb.dll!Owin.Loader.DefaultLoader.MakeDelegate.AnonymousMethod__b(Owin.IAppBuilder builder)   Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Owin.Loader.DefaultLoader.LoadImplementation.AnonymousMethod__0(Owin.IAppBuilder builder) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint.AnonymousMethod__0(Owin.IAppBuilder builder) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinAppContext.Initialize(System.Action<Owin.IAppBuilder> startup)  Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinBuilder.Build(System.Action<Owin.IAppBuilder> startup)  Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.InitializeBlueprint()    Unknown
    mscorlib.dll!System.Threading.LazyInitializer.EnsureInitializedCore<System.__Canon>(ref System.__Canon target, ref bool initialized, ref object syncLock, System.Func<System.__Canon> valueFactory) Unknown
    Microsoft.Owin.Host.SystemWeb.dll!Microsoft.Owin.Host.SystemWeb.OwinHttpModule.Init(System.Web.HttpApplication context) Unknown
    System.Web.dll!System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(System.IntPtr appContext, System.Web.HttpContext context, System.Reflection.MethodInfo[] handlers)  Unknown
    System.Web.dll!System.Web.HttpApplication.InitSpecial(System.Web.HttpApplicationState state, System.Reflection.MethodInfo[] handlers, System.IntPtr appContext, System.Web.HttpContext context) Unknown
    System.Web.dll!System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(System.IntPtr appContext, System.Web.HttpContext context)    Unknown
    System.Web.dll!System.Web.Hosting.PipelineRuntime.InitializeApplication(System.IntPtr appContext)   Unknown
    [AppDomain Transition]

Repro steps:
Run a website in IISExpress. Save the web.config and reload the site several times.
About 4-5 tries later of recycling the site it hangs.

I tried debugging the disassembled code and all I could see is the code jumping indefinitely between those two functions.

>   System.dll!System.Diagnostics.PerformanceCounterLib.GetPerformanceCounterLib(string machineName, System.Globalization.CultureInfo culture)  Unknown
>   System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown

This looks to me like a by-product of this bug:
http://connect.microsoft.com/VisualStudio/feedback/details/864245/multithreaded-bug-in-system-diagnostics-performancecounter

I'd like to have a config value to disable the creation of Performance Counters so I can avoid this problem if possible.

Source

corneliutusnea

Most helpful comment

See this blog post - we faced the same issue, in the OWIN Startup method, when creating a MemoryCache instance. The root cause is CultureInfo leaking from one AppDomain to another. A workaround has been identified, but the real issue lies within OWIN that should sanitize the culture before running the Startup method.

zpqrtbnk on 31 Oct 2015

👍3

All 33 comments

This helps on my machine, I found the solution for this when I had similar hangs with the MemoryCache implementation. Running it on a separate thread somehow bypasses the PerfCounter problem.
Hope this helps :)

var task=Task.Run(()=>app.MapSignalR());
            task.Wait(300);
//try again if it fails just to be sure ;)
            if (task.IsCanceled)  Task.Run(() => app.MapSignalR()).Wait(300);

JonasSyrstad on 23 Feb 2015

For now I found another solution. I implemented a dummy no-op IPerformanceCounterManager and registered it in the global host just before the SignalR gets mapped. This makes SR use my own performance counter manager:

var tempCounterManager = new TempPerformanceCounterManager();
GlobalHost.DependencyResolver.Register(typeof (IPerformanceCounterManager), () => tempCounterManager);

[....]

public class TempPerformanceCounterManager : IPerformanceCounterManager
{
    private readonly static PropertyInfo[] _counterProperties = GetCounterPropertyInfo();
    private readonly static IPerformanceCounter _noOpCounter = new NoOpPerformanceCounter();

    public TempPerformanceCounterManager()
    {
        foreach (var property in _counterProperties)
        {
            property.SetValue(this, new NoOpPerformanceCounter(), null);
        }
    }

    public void Initialize(string instanceName, CancellationToken hostShutdownToken)
    {
    }

    public IPerformanceCounter LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)
    {
        return _noOpCounter;
    }

    internal static PropertyInfo[] GetCounterPropertyInfo()
    {
        return typeof(TempPerformanceCounterManager)
            .GetProperties()
            .Where(p => p.PropertyType == typeof(IPerformanceCounter))
            .ToArray();
    }
    public IPerformanceCounter ConnectionsConnected { get; set; }
    public IPerformanceCounter ConnectionsReconnected { get; set; }
    public IPerformanceCounter ConnectionsDisconnected { get; set; }
    public IPerformanceCounter ConnectionsCurrentForeverFrame { get; private set; }
    public IPerformanceCounter ConnectionsCurrentLongPolling { get; private set; }
    public IPerformanceCounter ConnectionsCurrentServerSentEvents { get; private set; }
    public IPerformanceCounter ConnectionsCurrentWebSockets { get; private set; }
    public IPerformanceCounter ConnectionsCurrent { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentTotal { get; private set; }
    public IPerformanceCounter ConnectionMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ConnectionMessagesSentPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter ScaleoutMessageBusMessagesReceivedPerSec { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedTotal { get; private set; }
    public IPerformanceCounter MessageBusMessagesPublishedPerSec { get; private set; }
    public IPerformanceCounter MessageBusSubscribersCurrent { get; private set; }
    public IPerformanceCounter MessageBusSubscribersTotal { get; private set; }
    public IPerformanceCounter MessageBusSubscribersPerSec { get; private set; }
    public IPerformanceCounter MessageBusAllocatedWorkers { get; private set; }
    public IPerformanceCounter MessageBusBusyWorkers { get; private set; }
    public IPerformanceCounter MessageBusTopicsCurrent { get; private set; }
    public IPerformanceCounter ErrorsAllTotal { get; private set; }
    public IPerformanceCounter ErrorsAllPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionTotal { get; private set; }
    public IPerformanceCounter ErrorsHubResolutionPerSec { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationTotal { get; private set; }
    public IPerformanceCounter ErrorsHubInvocationPerSec { get; private set; }
    public IPerformanceCounter ErrorsTransportTotal { get; private set; }
    public IPerformanceCounter ErrorsTransportPerSec { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountTotal { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountOpen { get; private set; }
    public IPerformanceCounter ScaleoutStreamCountBuffering { get; private set; }
    public IPerformanceCounter ScaleoutErrorsTotal { get; private set; }
    public IPerformanceCounter ScaleoutErrorsPerSec { get; private set; }
    public IPerformanceCounter ScaleoutSendQueueLength { get; private set; }
}

internal class NoOpPerformanceCounter : IPerformanceCounter
{
    public string CounterName
    {
        get
        {
            return GetType().Name;
        }
    }
    public long Decrement()
    {
        return 0;
    }
    public long Increment()
    {
        return 0;
    }
    public long IncrementBy(long value)
    {
        return 0;
    }
    public long RawValue
    {
        get { return 0; }
        set { }
    }
    public void Close()
    {
    }
    public void RemoveInstance()
    {
    }
    public CounterSample NextSample()
    {
        return CounterSample.Empty;
    }
}

corneliutusnea on 23 Feb 2015

❤1 🎉1

Are you able to tell what the values of categoryName, counterName and instanceName were when it got stuck? Also, do you know if this happens for the same performance counter or different ones?

PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)

moozzyk on 23 Feb 2015

The same problem occurs randomly on Visual Studio 2015, I think is because the new diagnostic tool

alexandrereyes on 24 Jul 2015

I discovered the same behaviour on my development machine as well as on my Windows Azure Cloud Service machine. In both cases I am using the full IIS. On my development machine the cpu consumption of the w3wp.exe rises and never lowers again. The website then gets very unresponsive. It only seems to happen when there are existing SignalR connections and the AppPool and thereby the w3wp.exe are restarted. I attached my Visual Studio 2015 to the w3wp process and see that one thread hangs on the kine that call MapSignalR. The call stack tells me, that the method called some external code but i can't see any name or hint what it is doing.

sharpdevel on 12 Aug 2015

I had the same hang issue with my app (not related to SignalR at all) in perfcounter lib (I was using MemoryCache but that's the same story as MemoryCache uses perf counters at initialization - as others in this thread noted). There is a strange piece of code in the private System.dll's method PerformanceCounterLib:IsCustomCategory(string, string) that can possibly loop forever scanning the registry _if the current culture/LCID is not 9 -> english_. Here is what I did to fix the issue:

        CultureInfo ci = Thread.CurrentThread.CurrentCulture;
        var lcid = CultureInfo.CurrentCulture.Parent.LCID;
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfo(1033);
        }
        var myCache = new MemoryCache(MyCachePrefix, MyCacheConfig);
        if (lcid != 9)
        {
            Thread.CurrentThread.CurrentCulture = ci;
        }

smourier on 20 Aug 2015

I have the same problem in 2.2.0, Windows 10, VS 2015 and IIS Express. It happens after recycling IIS App Pool. This is my call stack:

[Managed to Native Transition]  
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name, bool writable) Unknown
mscorlib.dll!Microsoft.Win32.RegistryKey.OpenSubKey(string name)    Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.FindCustomCategory(string category, out System.Diagnostics.PerformanceCounterCategoryType categoryType) Unknown
System.dll!System.Diagnostics.PerformanceCounterLib.IsCustomCategory(string machine, string category)   Unknown
System.dll!System.Diagnostics.PerformanceCounter.InitializeImpl()   Unknown
System.dll!System.Diagnostics.PerformanceCounter.PerformanceCounter(string categoryName, string counterName, string instanceName, bool readOnly)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.LoadCounter(string categoryName, string counterName, string instanceName, bool isReadOnly)  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.SetCounterProperties()  Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Infrastructure.PerformanceCounterManager.Initialize(string instanceName, System.Threading.CancellationToken hostShutdownToken)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Microsoft.AspNet.SignalR.Hosting.HostDependencyResolverExtensions.InitializeHost(Microsoft.AspNet.SignalR.IDependencyResolver resolver, string instanceName, System.Threading.CancellationToken hostShutdownToken)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.UseSignalRMiddleware<Microsoft.AspNet.SignalR.Owin.Middleware.HubDispatcherMiddleware>(Owin.IAppBuilder builder, object[] args)   Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR.AnonymousMethod__0(Owin.IAppBuilder subApp)    Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, Microsoft.Owin.PathString pathMatch, System.Action<Owin.IAppBuilder> configuration) Unknown
Microsoft.Owin.dll!Owin.MapExtensions.Map(Owin.IAppBuilder app, string pathMatch, System.Action<Owin.IAppBuilder> configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, string path, Microsoft.AspNet.SignalR.HubConfiguration configuration)    Unknown
Microsoft.AspNet.SignalR.Core.dll!Owin.OwinExtensions.MapSignalR(Owin.IAppBuilder builder, Microsoft.AspNet.SignalR.HubConfiguration configuration) Unknown
...

I don't know how to find out what it is trying to do with registry but it is always waiting there. I had suspected that it was related to permissions and tried with running VS as administrator; but it didn't solve the problem.

alisabzevari on 19 Sep 2015

Like I said, it's a bug in this IsCustomCategory .NET's code here: http://referencesource.microsoft.com/#System/services/monitoring/system/diagnosticts/PerformanceCounterLib.cs,fb23e462a0395092,references when the current LCID/locale/language is not English, this code can go in a forever loop for some reason (not always). I you can force the culture to English before the code stack reaches this method, it should fix the problem.

smourier on 19 Sep 2015

❤1

I have the same issue.. Although only after having:

attached a debugger
stopped debugging
build, rebuild or recycled
verified that MapSignalR is the culprit (threads are looping forever)

Windows 10 (IIS 10), Visual Studio 2015 and SignalR 2.2.0

drphrozen on 22 Sep 2015

@smourier Thanks your solution worked for me.

alisabzevari on 22 Sep 2015

No official fix for this yet? :(

MojoDK on 24 Oct 2015

zpqrtbnk on 31 Oct 2015

👍3

Nice analysis. I would just add that for me the real issue is in the .NET framework (System.dll), not OWIN nor SignalR specifically. Everyone using PerformanceCounter directly or indirectly (in my case, I'm not using OWIN nor SignalR, just MemoryCache) has the problem. It's a quite big issue in fact. Don't know if Microsoft is aware of this BTW - I suppose their developers are mostly running with en English LCID so they don't see the problem often :-) ...

smourier on 31 Oct 2015

One could definitively argue that the problem lies within the PerformanceCounterLib class that should _not_ compare CultureInfo instances by reference.

zpqrtbnk on 31 Oct 2015

Of course, yes but this class is internal. But as soon as you use public one PerformanceCounter (and work with non English LCID) you can face the issue.

smourier on 31 Oct 2015

Thanks Jonas Syrstad. Really is the solution.

harveytriana on 19 Nov 2015

@zpqrtbnk - this is a great write up. I prepared a fix (#3609) based on your post. Do you mind taking a look?

moozzyk on 5 Dec 2015

Looks good to me.

zpqrtbnk on 7 Dec 2015

Fixed in 23e5b1b5bcceec097cc4e4e0f32fd00303ad8746

moozzyk on 7 Dec 2015

🎉1

Is it possible that this could this fix be applied at the Owin level (not sure where that repo lives)? This will fix the issue seen in SignalR but won't fix the issue for any other Owin based package

Shazwazza on 14 Dec 2015

When do you guys expect this update to be included in nuget? Last nuget version is 2.2.0 (jan 13th).

MojoDK on 15 Dec 2015

@Shazwazza - we working on this as we speak

moozzyk on 15 Dec 2015

Same problem here. Glad you guys already fixed it.
Waiting for the next version.
Thanks.

KimTranjan on 21 Dec 2015

@KimTranjan - the credits go to @smourier and @zpqrtbnk

moozzyk on 21 Dec 2015

Yeah, I did read the thread and the blog post, btw.
This community is great.

edit:

I'm using @zpqrtbnk's workaround and it's working perfectly.

KimTranjan on 21 Dec 2015

I can see the problem also. when we can see the update in nuget ?

thanks.

MosheL on 31 Jan 2016

I have the problem too. I tried using the Unofficial nuget package, but this messes up my packages. I'd rather use the official release. When will this be made official?

woutercx on 6 Feb 2016

I have SignalR version 2.2.1 from nuget and I'm using the solution from zpqrtbnk (https://www.zpqrtbnk.net/posts/appdomains-threads-cultureinfos-and-paracetamol) as well, but the problem still occurrs.
I'm using windows 10 with IIS 10

flori27 on 14 Sep 2016

@flori27 - 2.2.1 should have this workaround (https://github.com/SignalR/SignalR/commit/23e5b1b5bcceec097cc4e4e0f32fd00303ad8746). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?

moozzyk on 14 Sep 2016

Just to note - The problem is only on Windows 10, as I think. In Windows 7 or 2008R2 I do not see it.

MosheL on 14 Sep 2016

@MosheL - I did try Windows 10 (as well as Win 2008 R2) and could not repro this. Is there anything specific about your environment? One of the theories is that it happens if the OS/Thread is not using english culture. I did try non-english versions of both Win10 and Win2008R2 and still could not repro. Do you have any specific language settings on your Windows 10 machine where this happens?

moozzyk on 14 Sep 2016

I am using Hebrew-modified one, and also with web.config with culture (en-GB).

MosheL on 14 Sep 2016

Hi,

In IIS, I have 2 bindings for my website
ts.localhost.comhttp://ts.localhost.com (configured in VS to start on this URL and points to my localhost)
pcwork.no-ip.bizhttp://pcwork.no-ip.biz (points to my internal IP address)

Other local computers are connected to pcwork.no-ip.bizhttp://pcwork.no-ip.biz.
When I use this url, IIS hangs.
I don't have the problem ts.localhost.comhttp://ts.localhost.com, but other devices don't know this url, so I need to use pcwork.no-ip.bizhttp://pcwork.no-ip.biz

Op 14 sep. 2016, om 18:04 heeft Pawel Kadluczka <[email protected]notifications@github.com> het volgende geschreven:

@flori27https://github.com/flori27 - 2.2.1 should have this workaround (23e5b1bhttps://github.com/SignalR/SignalR/commit/23e5b1b5bcceec097cc4e4e0f32fd00303ad8746). The issue is that we have not been able to repro this issue ever so we took the fix hoping it will resolve the issue. If this is still occurring - how do I repro this? Is there anything special about your environment?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/SignalR/SignalR/issues/3414#issuecomment-247063874, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFIXr4eIqMTUTW9K5Zri-OLbKuA5oOqOks5qqBr6gaJpZM4DkCkp.

flori27 on 15 Sep 2016

Was this page helpful?

0 / 5 - 0 ratings