Stackexchange.redis: Timeouts on Azure Redis Cache StringGet calls (high qs and in values)

[SkyKicker]

We have a heavily used web service which uses StackExchange.Redis version 1.0.488 (latest on NuGet). This runs on an Azure Web Server talking to a Azure Redis Server in the same resource group. We are currently running a single web server instance and a C4 server and I am curious if the data in the error messages points to a particular constraint. Does the 'in' value of 65536 indicate to me that the library is not consuming the responses fast enough. The CPU % is low, so I don't see how that would be the case.

Here are a couple of the exceptions I have gotten on StringGet calls.

Timeout performing GET ed8839e5-c9bf-e511-80fb-3863bb3c1190, inst: 14, mgr: Inactive, err: never, queue: 2715, qu: 0, qs: 2715, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, IOCP: (Busy=175,Free=825,Min=8,Max=1000), WORKER: (Busy=0,Free=32767,Min=8,Max=32767), clientName: xxxxxxxxxxxxx at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 80
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1451

Timeout performing GET ed8839e5-c9bf-e511-80fb-3863bb3c1190, inst: 14, mgr: Inactive, err: never, queue: 2715, qu: 0, qs: 2715, qc: 0, wr: 0, wq: 0, in: 65536, ar: 0, IOCP: (Busy=175,Free=825,Min=8,Max=1000), WORKER: (Busy=0,Free=32767,Min=8,Max=32767), clientName: xxxxxxxxxxxxx at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 80
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags) in c:\TeamCity\buildAgent\work\3ae0647004edff78\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1451

[xqrzd]

I had a similar issue, you could try increasing syncTimeout in your config. By default it's only 1s, and when I was doing some large StringGet requests, it would occasionally timeout.

https://github.com/StackExchange/StackExchange.Redis/blob/master/Docs/Configuration.md

[SkyKicker]

I've got the timeout at 2500 ms and may try higher. Meanwhile, I'm going to invest some time into looking into using ServiceStack to access Redis.

[SkyKicker]

A little help from my friends at Microsoft did the trick. They pointed me to the post:

https://gist.github.com/JonCole/e65411214030f0d823cb

When busy thread count reaches min thread count for both .NET IOCP and Worker thread pools then .NET will throttle additional threads. This can lead to Redis client timeouts.

It turns out that the default values for minIoThreads and minWorkerThreads were too low which was evident in the error log "IOCP: (Busy=175,Free=825,Min=8,Max=1000)". After adding code into the Web Service global.aspx to set both minIoThreads and minWorkerThreads to 300, there have been no timeouts.

        int defaultMinWorkerThreads = 300;
        int defaultMinIoThreads = 300;
        string strDIo = ConfigurationManager.AppSettings["DefaultMinWorkerThreads"];
        string strDWo = ConfigurationManager.AppSettings["DefaultMinIoThreads"];
        int.TryParse(strDIo, out defaultMinIoThreads);
        int.TryParse(strDWo, out defaultMinWorkerThreads);
        ThreadPool.SetMinThreads(defaultMinWorkerThreads, defaultMinIoThreads);