Stackexchange.redis: RedisTimeoutException with 2.0.519

[dharmenbavaria]

We are using StackExchange.Redis very heavily in our ASP.NET core webapi application. We use https://github.com/imperugo/StackExchange.Redis.Extensions as an extension for serializing and other stuff. Since we were getting timeout in previous version before 1.2. We migrated to latest version since there were fix for timout but still that same issue ongoing.
Every now and then at some time we start getting redistimeout. We have no idea whats going on. We tried various approach recommended on timeout link coming in exception:
1) Increaded ThreadPool.MinThread
2) Used ConnectionPooling for multiplexer
3) We monitor redis everything is fine CPU, network etc.

When it happens we have series of timeout exceptions. Some with output buffer and some normal one.

Timeout awaiting response (5094ms elapsed, timeout is 5000ms), inst: 0, qs: 10, in: 65536, mgr: 10 of 10 available, IOCP: (Busy=26,Free=974,Min=8,Max=1000), WORKER: (Busy=3,Free=32764,Min=8,Max=32767), v: 2.0.513.63329 
Timeout awaiting response (5063ms elapsed, timeout is 5000ms), inst: 0, qs: 4, in: 47355, mgr: 10 of 10 available, IOCP: (Busy=40,Free=960,Min=8,Max=1000), WORKER: (Busy=34,Free=32733,Min=8,Max=32767), v: 2.0.513.63329 
The timeout was reached before the message could be written to the output buffer, and it was not sent (5000ms, inst=3, qs=3, in=10, active=HMSET), inst: 3, qs: 3, in: 10, mgr: 10 of 10 available, IOCP: (Busy=36,Free=964,Min=8,Max=1000), WORKER: (Busy=34,Free=32733,Min=8,Max=32767), v: 2.0.513.63329

Image

Several output buffer exception.
Serveral redis timeout exception.
For some sepecific instances CPU going higher to 80-90%.

We dont't have thread pool starvation for sure because we are monitoring threads as well.

I would really appreciate if someone could suggest what is going wrong here and is there any way we can prevent this thing to happen.

[WeihanLi]

similar question, my work thread and iocp is not busy

An unhandled exception has occurred while executing the request
StackExchange.Redis.RedisTimeoutException: Timeout performing EXISTS vote:3:vote-items (5000ms), inst: 0, qs: 1459, in: 0, serverEndpoint: Unspecified/zy-dev01.redis.cache.chinacloudapi.cn:6380, mgr: 10 of 10 available, clientName: DESKTOP-2169HK5, IOCP: (Busy=0,Free=1000,Min=128,Max=1000), WORKER: (Busy=1460,Free=31307,Min=2048,Max=32767), v: 2.0.519.65453 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2197
   at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54
   at StackExchange.Redis.RedisDatabase.KeyExists(RedisKey key, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 649

[zhangruiskyline]

I have similar question, according to https://github.com/StackExchange/StackExchange.Redis/blob/master/docs/Timeouts.md

2.0+ lib have a dedicated threadpool to handle all redis connection thread, so do we still need to manually set minimum thread number in order to prevent timeout?

[zhangruiskyline]

Also in general, it seems very hard to manage the thread in this, if we start from default thread number based on core number, the minimum thread number will be low, then it will easily scale to high worker threads and cause timeout, but if we intentionally set high minimum thread number, high contention and context switch may be a issue

Any advice on this?

[evhfla-zz]

Having the same issue

[ispysoftware]

here's our redis timeout server errors - v1.2.6 working - upgrade to latest and then rollback. No other code changes:

[redsquare]

@ispysoftware ssl connection?

[WeihanLi]

I'm using with ssl connection @redsquare

[ispysoftware]

Yes, using SSL. This fixed it for us:
https://stackoverflow.com/questions/29603890/azure-redis-cache-pool-of-connectionmultiplexer-objects

[ChrisMcKee]

public static int POOL_SIZE = 100;
private static readonly Object lockPookRoundRobin = new Object();
private static Lazy<Context>[] lazyConnection = null;

//Static initializer to be executed once on the first call
private static void InitConnectionPool()
{
    lock (lockPookRoundRobin)
    {
        if (lazyConnection == null) {
             lazyConnection = new Lazy<Context>[POOL_SIZE];
        }


        for (int i = 0; i < POOL_SIZE; i++){
            if (lazyConnection[i] == null)
                lazyConnection[i] = new Lazy<Context>(() => new Context("YOUR_CONNECTION_STRING", new CachingFramework.Redis.Serializers.JsonSerializer()));
        }
    }
}

private static Context GetLeastLoadedConnection()
{
    //choose the least loaded connection from the pool
    /*
    var minValue = lazyConnection.Min((lazyCtx) => lazyCtx.Value.GetConnectionMultiplexer().GetCounters().TotalOutstanding);
    var lazyContext = lazyConnection.Where((lazyCtx) => lazyCtx.Value.GetConnectionMultiplexer().GetCounters().TotalOutstanding == minValue).First();
    */

    // UPDATE following @Luke Foust comment below
    Lazy<Connection> lazyContext;

    var loadedLazys = lazyConnection.Where((lazy) => lazy.IsValueCreated);
    if(loadedLazys.Count()==lazyConnection.Count()){
        var minValue = loadedLazys.Min((lazy) => lazy.Value.TotalOutstanding);
        lazyContext = loadedLazys.Where((lazy) => lazy.Value.TotalOutstanding == minValue).First();
    }else{
        lazyContext = lazyConnection[loadedLazys.Count()];
    }
    return lazyContext.Value;
}

private static Context Connection
{
    get
    {
        lock (lockPookRoundRobin)
        {
            return GetLeastLoadedConnection();
        }
    }
}

public RedisCacheService()
{
    InitConnectionPool();
}

Linked runnable pool sans connection - https://dotnetfiddle.net/UCJFeA

The code from that SO answer.
We were seeing SSL errors; I'm not sure pooling the pooled multiplexer is anything more than a sticky plaster over an underlying issue.

[ispysoftware]

Maybe it's a TLS 1.2 issue (https://stackoverflow.com/questions/28286086/default-securityprotocol-in-net-4-5) - are you saying that setting SSL to false should solve it?

[redsquare]

Its not a 1.2 issue, happens on azure over 1.0, ssl false fixes it which isn't great

[mgravell]

We are aware of a problem here (relating to a race condition in the write-lock); we're dogfooding a fix locally, and hope to deploy an update in the next day or two.

[mgravell]

The combined improvements in 2.0.588 and 2.0.571 should, we believe, address a wide range of timeout and performance problems. I'm closing all timeout issues as "believed fixed in 2.0.588"; if you still see problems, please report them as new issues so we can start investigation with fresh eyes.

[alireza267]

@mgravell we still have the same problem in 2.0.601 !

[jeroenpot]

We also still faced the problem, so decided to use another library.

[humayun-ahmed]

We also still faced the problem, so decided to use another library.

which library are you using now?

[jeroenpot]

We also still faced the problem, so decided to use another library.

which library are you using now?

EasyCaching.CSRedis
https://github.com/dotnetcore/EasyCaching
I did some basic load-testing (parallel async adding / getting a lot of data), and it was easy to get the timeout exception with StackExchange.Redis. Didn't manage to get the issue (or simular problems) with easyCaching lib.

[redsquare]

Note that EasyCaching uses Stack under the hood!

[jeroenpot]

Note that EasyCaching uses Stack under the hood!

Use EasyCaching.CSRedis not EasyCaching.Redis

[humayun-ahmed]

Is EasyCaching.CSRedis supports cluster enabled redis?

On Fri, Oct 25, 2019 at 3:06 PM Jeroen Pot [e-mail] wrote:

Note that EasyCaching uses Stack under the hood!

Use EasyCaching.CSRedis not EasyCaching.Redis

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/1036?email_source=notifications&email_token=AAJ47ASRZQNJK43RDHLZILDQQLVL3A5CNFSM4GOT6Q22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECIJGSI#issuecomment-546345801,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJ47ASRLBAZMQ44YIMGRSDQQLVL3ANCNFSM4GOT6Q2Q
.

--
Humayun Ahmed

[zhanghaiboshiwo]

I also met the same problem in 2.0.601. Do you have a solution now?