StackExchange.Redis.RedisTimeoutException

[Iamcerba]

I'm using 1.2.6 version of the package and often see the following exceptions in logs while connecting to Azure RedisCaches. Any ideas what may cause such errors?

StackExchange.Redis.RedisTimeoutException: Timeout performing EXISTS controllers.usercontroller-getusersscroll-userIds=&userIds=&skip=0&take=2&status=Active, Inactive&orderBy=0&sortDirection=Desc:application/json; charset=utf-8:Bearer , inst: 0, mgr: Inactive, err: never, queue: 1769, qu: 1769, qs: 0, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: <...>, serverEndpoint: Unspecified/<...>.redis.cache.windows.net:6380, keyHashSlot: 2692, IOCP: (Busy=0,Free=1000,Min=250,Max=1000), WORKER: (Busy=1,Free=32766,Min=250,Max=32767)

[vipwangtian]

I had the same problem
StackExchange.Redis.RedisTimeoutException: Timeout performing EVAL, inst: 105, mgr: ExecuteSelect, err: never, queue: 28, qu: 2, qs: 26, qc: 0, wr: 1, wq: 1, in: 10, ar: 0, clientName: MANGOSQLSERVER, serverEndpoint: Unspecified/171.redis.517.cn:6381, keyHashSlot: 16346, IOCP: (Busy=0,Free=3200,Min=64,Max=3200), WORKER: (Busy=137,Free=3063,Min=512,Max=3200) (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)

[tocalai]

I used version 1.2.6 also encountered the same issue:
StackExchange.Redis.RedisTimeoutException
Timeout performing RPUSH xxxx:xxxx:xxxx:ParentJob:9ec051d6-48fa-4a46-9ff1-e14f0c26ce0c:Site:1, inst: 303, mgr: ExecuteSelect, err: never,
queue: 1, qu: 1, qs: 0, qc: 0, wr: 0, wq: 1, in: 0, ar: 0,
clientName: xxxx-PC, serverEndpoint: Unspecified/localhost:6379,
keyHashSlot: 4935, IOCP: (Busy=0,Free=1000,Min=100,Max=1000), WORKER: (Busy=2,Free=8189,Min=100,Max=8191)
(Please take a look at this article for some common client-side issues that can cause timeouts:
http://stackexchange.github.io/StackExchange.Redis/Timeouts)

Looks into the article for more help, seems that we do not hit the limit of Busy > Min scenario
The exception not happened every time, but usually occurred in amount of data(> 10000 records) under the multi-thread condition for access the Redis server.

Any suggestion for this?

[vitali-shcharbin]

Similar issue with 1.2.6 version, that happens from time to time:

StackExchange.Redis.RedisTimeoutException :
Timeout performing PUBLISH xxxxxxxxxxxxx, inst: 1, mgr: ExecuteSelect, err: never, queue: 2, qu: 0, qs: 2, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: XXXX, serverEndpoint: IP:6379,
IOCP: (Busy=1,Free=999,Min=6,Max=1000),
WORKER: (Busy=1,Free=32766,Min=6,Max=32767)
(Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)

[ozgurrgul]

I also encounter same problem in .NET Core 2:

Timeout performing HGETALL xxxxxxxx, inst: 1, queue: 6, qu: 0, qs: 6, qc: 0, wr: 0, wq: 0, in: 341, ar: 0, clientName: , serverEndpoint: Unspecified/localhost:6379, keyHashSlot: 11105 (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)

[georgechr]

Same issue, with version 1.2.6 this happens from time to time

StackExchange.Redis.RedisTimeoutException: Timeout performing HMSET , inst: 1, queue: 28, qu: 0, qs: 28, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: , serverEndpoint: , keyHashSlot: 6178 (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)

[EminemJK]

I also encounter same problem In V1.2.6，and then I download the source code and see what happens here.

` internal T ExecuteSyncImpl(Message message, ResultProcessor processor, ServerEndPoint server)
{
if (isDisposed) throw new ObjectDisposedException(ToString());

        if (message == null) // fire-and forget could involve a no-op, represented by null - for example Increment by 0
        {
            return default(T);
        }

        if (message.IsFireAndForget)
        {
            TryPushMessageToBridge(message, processor, null, ref server);
            Interlocked.Increment(ref fireAndForgets);
            return default(T);
        }
        else
        {
            var source = ResultBox<T>.Get(null);

            lock (source)
            {
                if (!TryPushMessageToBridge(message, processor, source, ref server))
                {
                    throw ExceptionFactory.NoConnectionAvailable(IncludeDetailInExceptions, IncludePerformanceCountersInExceptions, message.Command, message, server, GetServerSnapshot());
                }

                if (Monitor.Wait(source, TimeoutMilliseconds))
                {
                    Trace("Timeley response to " + message);
                }
                else
                {
                    Trace("Timeout performing " + message);
                    Interlocked.Increment(ref syncTimeouts);
                    string errMessage;
                    List<Tuple<string, string>> data = null;
                    if (server == null || !IncludeDetailInExceptions)
                    {
                        errMessage = "Timeout performing " + message.Command.ToString();
                    }
                    else
                    {
                        var mgrState = socketManager.State;
                        var lastError = socketManager.LastErrorTimeRelative();
                        var sb = new StringBuilder("Timeout performing ").Append(message.CommandAndKey);
                        data = new List<Tuple<string, string>> { Tuple.Create("Message", message.CommandAndKey) };
                        void add(string lk, string sk, string v)
                        {
                            data.Add(Tuple.Create(lk, v));
                            sb.Append(", ").Append(sk).Append(": ").Append(v);
                        }

                        int queue = server.GetOutstandingCount(message.Command, out int inst, out int qu, out int qs, out int qc, out int wr, out int wq, out int @in, out int ar);
                        add("Instantaneous", "inst", inst.ToString());
                        add("Manager-State", "mgr", mgrState.ToString());
                        add("Last-Error", "err", lastError);
                        add("Queue-Length", "queue", queue.ToString());
                        add("Queue-Outstanding", "qu", qu.ToString());
                        add("Queue-Awaiting-Response", "qs", qs.ToString());
                        add("Queue-Completion-Outstanding", "qc", qc.ToString());
                        add("Active-Writers", "wr", wr.ToString());
                        add("Write-Queue", "wq", wq.ToString());
                        add("Inbound-Bytes", "in", @in.ToString());
                        add("Active-Readers", "ar", ar.ToString());

                        add("Client-Name", "clientName", ClientName);
                        add("Server-Endpoint", "serverEndpoint", server.EndPoint.ToString());
                        var hashSlot = message.GetHashSlot(ServerSelectionStrategy);
                        // only add keyslot if its a valid cluster key slot
                        if (hashSlot != ServerSelectionStrategy.NoSlot)
                        {
                            add("Key-HashSlot", "keyHashSlot", message.GetHashSlot(this.ServerSelectionStrategy).ToString());
                        }
                        int busyWorkerCount = GetThreadPoolStats(out string iocp, out string worker);
                        add("ThreadPool-IO-Completion", "IOCP", iocp);
                        add("ThreadPool-Workers", "WORKER", worker);
                        data.Add(Tuple.Create("Busy-Workers", busyWorkerCount.ToString()));

                        if (IncludePerformanceCountersInExceptions)
                        {
                            add("Local-CPU", "Local-CPU", GetSystemCpuPercent());
                        }
                        sb.Append(" (Please take a look at this article for some common client-side issues that can cause timeouts: ");
                        sb.Append(timeoutHelpLink);
                        sb.Append(")");
                        errMessage = sb.ToString();
                        if (StormLogThreshold >= 0 && queue >= StormLogThreshold && Interlocked.CompareExchange(ref haveStormLog, 1, 0) == 0)
                        {
                            var log = server.GetStormLog(message.Command);
                            if (string.IsNullOrWhiteSpace(log)) Interlocked.Exchange(ref haveStormLog, 0);
                            else Interlocked.Exchange(ref stormLogSnapshot, log);
                        }
                    }
                    var timeoutEx = ExceptionFactory.Timeout(IncludeDetailInExceptions, errMessage, message, server);
                    timeoutEx.HelpLink = timeoutHelpLink;

                    if (data != null)
                    {
                        foreach (var kv in data)
                        {
                            timeoutEx.Data["Redis-" + kv.Item1] = kv.Item2;
                        }
                    }
                    throw timeoutEx;
                    // very important not to return "source" to the pool here
                }
            }
            // snapshot these so that we can recycle the box
            ResultBox<T>.UnwrapAndRecycle(source, true, out T val, out Exception ex); // now that we aren't locking it...
            if (ex != null) throw ex;
            Trace(message + " received " + val);
            return val;
        }
    }`

I try to extend the TimeoutMilliseconds to solve it, it seems that the number of problems has decreased.

[aronsimpletix]

For the past 18 hrs I have been getting this error like too.
I am using 1.2.4
About to try 1.2.6 now

[JonCole]

A RedisTimeoutException being thrown does not necessarily constitute a bug in StackExchange.Redis nor does it necessarily mean that there is a bug in the server either. More often than not, it means that some computing limitation on the client or the server has been hit. Any of the issues outlined in the following links cause timeouts and they are all things that the application developer has to fix (for example: by scaling the client or server to a more powerful VM):

Diagnosing Client Issues
Diagnosing Server Issues

[zsy619]

StackExchange.Redis.RedisTimeoutException: Timeout performing GETBIT client:registered_accounts, inst: 35, queue: 1532, qu: 0, qs: 1532, qc: 0, wr: 0, wq: 0, in: 6474, ar: 0, clientName: , serverEndpoint: 192.168.1.8:6379, keyHashSlot: 2621 (Please take a look at this article for some common client-side issues that can cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\RedisBase.cs:line 81
at StackExchange.Redis.RedisDatabase.StringGetBit(RedisKey key, Int64 offset, CommandFlags flags) in c:\code\StackExchange.Redis\StackExchange.Redis\StackExchange\Redis\RedisDatabase.cs:line 1675
at TxSms.Redis.RedisHelper.BitGet(RedisKey key, Int64 offset) in D:\腾信集团\TxSms.Common\src\TxSms.Redis\RedisHelper.cs:line 943
at System.Linq.Enumerable.AllTSource
at TxSms.API.Services.SendMessage.d__5.MoveNext() in D:\腾信集团\TxSms.MP\src\TxSms.API\Services\SendMessage.cs:line 145
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at TxSms.API.Controllers.GroupMsgController.d__1.MoveNext() in D:\腾信集团\TxSms.MP\src\TxSms.API\Controllers\GroupMsgController.cs:line 53

[zousq19910623]

so ,is this problem have been solved??

[eltiare]

I get this error when I'm debugging. It's incredibly annoying and wastes a lot of time.

[JonCole]

Not all timeouts are caused by a single root issue. In this thread (e.g. the first timeout posted originally), I would say that the real issue is the "qu: 1769" entry in the timeout error message. This seems to indicate that the writer thread, which takes requests and pushes them onto the network, is dead. The other timeout error messages posted by others on this issue are likely caused by some other issue.

I have seen very few customers run into this "writer thread broken" issue, but it does happen. The workaround until the root cause is figured out would be to re-create the ConnectionMultiplexer object.

[eltiare]

I think it's because the debugger suspends execution of everything until I continue on. If I'm studying the current state for a minute or so and then continue, the error happens. If it's only a few seconds it does not.

[JonCole]

@eltiare, that is likely true for your specific case, but I don't think that is what this original issue was filed about. For debugger case, your only option that I can think of is to use a longer timeout when debugging.

[NickCraver]

We could possibly ignore all timeouts if Debugger.IsAttached, but...that has problems on the flip side of never seeing a timeout while debugging. I'm not sure that's a good way to go. To be fair, this happens for pretty much anything. For example, if you debug while a SQL query runs, you'll see the same timeout behavior.

[kierenj]

We get this too - I agree with Nick on it though. We just increase the timeout to a lot higher on our dev configs

[NickCraver]

Got some Twitter feedback here: let's add more text to the error message if Debugger.IsAttached at least, to help people out. It's not awesome, but it's better than nothing.

[eltiare]

Thanks for the tip to increase the timeout. Saved me a lot of hair pulling.

[glassmountains]

I have the same problem, but I'm fairly new to programming. Where can I increase the timeout? I already had a lot of hair pulling. Thanks!

[JonCole]

@glassmountains Take a look at ConfigurationOptions.SyncTimeout

[trimbleAdam]

I'm trying to make use of this lib within a .NET core 2.1 app, and the stack dump doesn't make ANY sense, as the app is running in a Debian container.

What is the business with this hard-coded c:\code\ fooery? Do I need to manually compile a package for the linux-x64 RID?

Any advice is greatly appreciated.

[mgravell]

Path information in a pdb relates to the build machine, not the executing
machine. This is normal. What is the actual stack trace?

On Thu, 14 Jun 2018, 21:43 trimbleAdam, [e-mail] wrote:

I'm trying to make use of this lib within a .NET core 2.1 app, and the
stack dump doesn't make ANY sense, as the app is running in a Debian
container.

What is the business with this hard-coded c:\code\ fooery? Do I need to
manually compile a package for the linux-x64 RID?

Any advice is greatly appreciated.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/760#issuecomment-397431910,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDsCN_4h6MAq8DfSI06BQBnaex5HHJks5t8srTgaJpZM4RJ3Xv
.

[mgravell]

Closing; please see: https://github.com/StackExchange/StackExchange.Redis/issues/871

[JedatKinports]

Could replacing the Connect() with the ConnectAsync() method fix this?