Stackexchange.redis: Timeout in Redis

[swapnashinde]

We are getting below errors after updating the StackExchange version to 2.0.601.
Could you please check.

Exception=StackExchange.Redis.RedisTimeoutException: Timeout performing GET (10000ms), next: PING, inst: 0, qu: 0, qs: 3, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: Unspecified/:6379, mgr: 10 of 10 available, clientName: , IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=1,Free=2046,Min=2,Max=2047), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT
at StackExchange.Redis.RedisDatabase.StringGet(RedisKey key, CommandFlags flags)

[swapnashinde]

Could you please help as we are getting this error in production.

[swapnashinde]

We are also getting below error
StackExchange.Redis.RedisConnectionException: No connection is available to service this operation: SETEX DBStreamingStatus; IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=0,Free=2047,Min=4,Max=2047), Local-CPU: n/a
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.StringSet(RedisKey key, RedisValue value, Nullable`1 expiry, When when, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 2407

[drakefang]

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[sampsonye]

we get the same issue,wait for help！

[oshvartz]

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[opendomain]

We are also having similar issues.
We have tried to change the size of the cache, moving to smaller object stored in the cache, asnyc/await, thread blocks, and serializing to JSON - nothing works.

[pengweiqhca]

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

85 > 8 and 100 > 8

[pengweiqhca]

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

20 > 4

[philiahe]

same problem

[MatanShabtay]

Has anyone try downgrading back to 1.2.6 ?
In all this time I only heard that version 2.x.xis far more stable than 1.x.x but it seems to be the opposite.

[Syncworx]

I was planning to migrate to the latest version. Should I stop it? I was getting CPU Spikes using version 1.2.6 when we have high traffic and the reason was related to Socket Manger.WriteToAllQueue. We are dead in the water now?

[liu-yfeng]

same problem.
but If we deploy server on windows , it's ok;
if we deploy on docker, there're lots of timeout exception.

Timeout performing EXISTS (5000ms), next: EXISTS test:insert:herokillrecord, inst: 5, qu: 0, qs: 23, aw: False, rs: ReadAsync, ws: Idle, in: 3885, in-pipe: 0, out-pipe: 45, serverEndpoint: 192.168.1.161:6380, mgr: 9 of 10 available, clientName: 5f42773d6dc1, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=32747,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

in 'xxx.runtimeconfig.json' file add
"System.Threading.ThreadPool.MinThreads": 100,
"System.Threading.ThreadPool.MaxThreads": 1000
like this:
{
"runtimeOptions": {
"tfm": "netcoreapp2.2",
"framework": {
"name": "Microsoft.AspNetCore.App",
"version": "2.2.0"
},
"configProperties": {
"System.GC.Server": true,
"System.Threading.ThreadPool.MinThreads": 100,
"System.Threading.ThreadPool.MaxThreads": 1000
}
}
}

[witskeeper]

i think StackExchange.Redis need warmup before handle high rate request.

i tested 2.x(2.0.601 & 2.0.513) in docker using jmeter

``` Docker file
FROM microsoft/dotnet:2.2-aspnetcore-runtime AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/dotnet:2.2-sdk AS build
WORKDIR /src
COPY ["RedisTest/RedisTest.csproj", "RedisTest/"]
RUN dotnet restore "RedisTest/RedisTest.csproj"
COPY . .
WORKDIR "/src/RedisTest"
RUN dotnet build "RedisTest.csproj" -c Release -o /app

FROM build AS publish
RUN dotnet publish "RedisTest.csproj" -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "RedisTest.dll"]

Test Code

[HttpPost]
public ActionResult Post([FromServices]IDatabase db)
{
var key = Guid.NewGuid().ToString();
db.StringSet(key, key);
return db.StringGet(key).ToString();
}

Setting of Test

number of thread : 20
Ramp-up Period : 1
```

if i start test just after i start docker container
i will get timeout error.

if i stop test and restart test.
timeout error never show up.

if i restart docker container and set Ramp-up Period to 100
timeout never show up.

if i stop test , set Ramp-up Period to 1 and wait for 20 min, then restart test.
i got timeout error.

so ,i think StackExchange.Redis need warmup before handle high rate request.
and it will cooling down if no request came in.

[Vamp282]

Same problem.
version 2.0.601

[marafiq]

What is the size of data, you are all trying to store? I have reduced my time outs by compressing the data, i store on Redis because your network pipe speed also matters.
Always set minIoThreads to at least 50 per processor.

[Vamp282]

Fixed. Increased for the min size of minWorker and minIOC.
https://gist.github.com/JonCole/e65411214030f0d823cb

[milovidov983]

Hi
we have the same problem - on high rate of Reads .net 4.6.2 with one master and 2 read replicas
StackExchange.Redis.RedisTimeoutException: Timeout performing GET (5000ms), next: GET 4CE01B05F4970DAC31C6DC06CA4D75, inst: 4, qu: 0, qs: 161, aw: False, rs: ReadAsync, ws: Idle, in: 61120, in-pipe: 168, out-pipe: 934, serverEndpoint: XXXXX, mgr: 10 of 10 available, clientName: XXXXX, IOCP: (Busy=85,Free=915,Min=8,Max=1000), WORKER: (Busy=100,Free=32667,Min=8,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

85 > 8 and 100 > 8

Sorry, I'm have same problem. My case:
IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=7,Free=32760,Min=2,Max=32767)
As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

Full exception:

! StackExchange.Redis.RedisTimeoutException: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5000ms elapsed, timeout is 5000ms), command=GET, next: GET CRM.UsersMainService.GetUserInfoAsync.[email protected], inst: 0, qu: 0, qs: 4, aw: False, rs: ReadAsync, ws: Idle, in: 1168, in-pipe: 0, out-pipe: 0, serverEndpoint: 5.178.85.30:7001, mgr: 10 of 10 available, clientName: f0b7b81f5ce5, PerfCounterHelperkeyHashSlot: 16089, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=7,Free=32760,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts) (Most recent call last)

[pengweiqhca]

As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

C# ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more

[milovidov983]

As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it?

ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more

I know about this, but I worry that Microsoft itself does not recommend setting arbitrary values....

[mgravell]

Synopsis: "it is chewing through an inbound stream with 161 replies needed
and 61k of data to look at; the next reply expected is the reply to the GET
shown"

Now; this doesn't tell us everything, unfortunately; it could be that this
"GET" is a huge oversized blob that is causing the stall, or it could be
that something earlier caused a stall, and it is just trying to catch up
when it ran out of time. But: take a look at what that key holds to see
whether there's anything alarming. You should also look at the server-side
logs to see if the server is taking a long time to process anything; see
"SLOWLOG GET".

Ignore the "busy workers" - that shouldn't matter here.

On Wed, 28 Aug 2019 at 19:18, Bullock [e-mail] wrote:

As you can see, the number of the busy workers is greater than the minimum
value 7 > 2. Could this be the cause of the error? How does this affect
the library? And what should I do to fix it?

ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more

I know about this, but I worry that Microsoft itself does not recommend
setting arbitrary values....

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/1150?email_source=notifications&email_token=AAAEHME2BKSHPS5I4DOOS33QG26OLA5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MAQ2A#issuecomment-525863016,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAEHMHLX6XS2VGZJ56EKTDQG26OLANCNFSM4HOLXHBQ
.

--
Regards,

Marc

[milovidov983]

Synopsis: "it is chewing through an inbound stream with 161 replies needed and 61k of data to look at; the next reply expected is the reply to the GET shown" Now; this doesn't tell us everything, unfortunately; it could be that this "GET" is a huge oversized blob that is causing the stall, or it could be that something earlier caused a stall, and it is just trying to catch up when it ran out of time. But: take a look at what that key holds to see whether there's anything alarming. You should also look at the server-side logs to see if the server is taking a long time to process anything; see "SLOWLOG GET". Ignore the "busy workers" - that shouldn't matter here.
…
On Wed, 28 Aug 2019 at 19:18, Bullock @.*> wrote: As you can see, the number of the busy workers is greater than the minimum value 7 > 2. Could this be the cause of the error? How does this affect the library? And what should I do to fix it? ThreadPool.SetMinThreads(int workerThreads, int completionPortThreads); //100 or more I know about this, but I worry that Microsoft itself does not recommend setting arbitrary values.... — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1150?email_source=notifications&email_token=AAAEHME2BKSHPS5I4DOOS33QG26OLA5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MAQ2A#issuecomment-525863016>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAEHMHLX6XS2VGZJ56EKTDQG26OLANCNFSM4HOLXHBQ .
-- Regards, Marc

Thanks for your reply. If I understand you correctly, the problem may be the size of the saved object? But all the objects that we save have a maximum size of 100 kilobytes

[mgravell]

Ok. But at a certain level of throughput, that might still be enough to
cause saturation issues on the pipe. We're investigating options for
providing concurrent independent pipes. I also can't know for sure, just on
this, whether this is indeed the cause. Stalls are very hard to pin down,
unfortunately. We need to be a little cautious as there's overheads in
adding more and more tracing to pinpoint it, that can then actively
contribute to causing the exact problem.

On Wed, 28 Aug 2019, 19:37 Bullock, [e-mail] wrote:

Synopsis: "it is chewing through an inbound stream with 161 replies needed
and 61k of data to look at; the next reply expected is the reply to the GET
shown" Now; this doesn't tell us everything, unfortunately; it could be
that this "GET" is a huge oversized blob that is causing the stall, or it
could be that something earlier caused a stall, and it is just trying
to catch up when it ran out of time. But: take a look at what that key
holds to see whether there's anything alarming. You should also look at the
server-side logs to see if the server is taking a long time to process
anything; see "SLOWLOG GET". Ignore the "busy workers" - that shouldn't
matter here.
… <#m_-8115909219667388450_>
On Wed, 28 Aug 2019 at 19:18, Bullock @.*> wrote: As you can see, the
number of the busy workers is greater than the minimum value 7 > 2.
Could this be the cause of the error? How does this affect the library? And
what should I do to fix it? ThreadPool.SetMinThreads(int workerThreads, int
completionPortThreads); //100 or more I know about this, but I worry that
Microsoft itself does not recommend setting arbitrary values.... — You are
receiving this because you are subscribed to this thread. Reply to this
email directly, view it on GitHub <#1150
https://github.com/StackExchange/StackExchange.Redis/issues/1150?email_source=notifications&email_token=AAAEHME2BKSHPS5I4DOOS33QG26OLA5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MAQ2A#issuecomment-525863016>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAEHMHLX6XS2VGZJ56EKTDQG26OLANCNFSM4HOLXHBQ
.
-- Regards, Marc

Thanks for your reply. If I understand you correctly, the problem may be
the size of the saved object? But all the objects that we save have a
maximum size of 100 kilobytes

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/1150?email_source=notifications&email_token=AAAEHMFBWQ3BMQCHWWHCV2DQG3AU3A5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5MCHUA#issuecomment-525870032,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAEHMGKB76MX7A2IWBCEALQG3AU3ANCNFSM4HOLXHBQ
.

[placidseven]

should I do to f

Did you ever get a response? I'm having timeout issues and seeing similar stats:

"WORKER: (Busy=5,Free=8186,Min=2,Max=8191"

I have a feeling this is being caused by queries the developers wrote are returning too much data, but I don't know how to prove it yet.

[timavaza]

Never really solved.. but i have determined that the naive examples often
cited thst create lots of groups and add people to groups is a bad idea..
as someone's browser connecting/disconnecting causes a lot of signalr redis
traffic as it re-sets up all the group memberships etc across all nodes.

I think the best appproach is to only have one group per user (that user's
user id) and whenever you need to send messages just use your other
database/cache to manage group memberships and send direct messages out to
all the appropriate users.

We not only send the absolute bare minimum data over signalr (e.g. entity
id that got updated) and the clients can go and refresh their current view
as needed over standard https ajax.

Also implement an exponential backoff algorithm on your clients, so they
don't take your servers down when you take a server out of your load
balancer and 000's of clients try to reconnect.. (and randomise those
reconnection delays )

On Tue, 22 Oct 2019, 20:32 placidseven, [e-mail] wrote:

should I do to f

Did you ever get a response? I'm having timeout issues and seeing similar
stats:

"WORKER: (Busy=5,Free=8186,Min=2,Max=8191"

I have a feeling this is being caused by queries the developers wrote are
returning too much data, but I don't know how to prove it yet.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/1150?email_source=notifications&email_token=AJ2SWSLTOMO5OHAEOT5UTE3QP5BNJA5CNFSM4HOLXHB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6X3LA#issuecomment-545095084,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AJ2SWSMSNQAJBK3MYTSNZYDQP5BNJANCNFSM4HOLXHBQ
.

[xiaoMaPrincess]

It is said that using all asynchronous methods can solve this problem.
We are ready to change csredis.
https://github.com/2881099/csredis

[NickCraver]

We're exploring a theory on a now-known cause and could use your help. For anyone experiencing issues on a .NET Framework application - are you in either of the following cases?

.NET Framework Web Projects

In web.config, either explicitly setting

<add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" />

or, something less than 4.5 in <httpRuntime> (also web.config):

<httpRuntime targetFramework="4.x" />

.NET Framework Non-web Projects

In app.config, something less than 4.5 in <supportedRuntime>:

<configuration>
    <startup> 
        <supportedRuntime version="v4.x" sku=".NETFramework,Version=v4.x"/>
    </startup>
</configuration>

To clarify: it doesn't matter what your .csproj (or other) target framework is for this case. These .config flags define runtime quirk behavior and alter how threads are handled - resulting in some of our should-be-dedicated bits being stolen and our queue indefinitely hung in a bad state.

If this matches anyone here, it'd be _hugely_ helpful to know. @mgravell is working on a workaround for this scenario.

[chrisdunne]

@NickCraver

Having the same intermittent timeout issues, using .NET Framework web project, however UseTaskFriendlySynchronizationContext is not explicitly set and the httpruntime isn't 4.x

[chrisdunne]

As mentioned by @drakefang I'm using Docker, that's the only commonality I can see in this thread.

[longjxchina]

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.

test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

[pengweiqhca]

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.

test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

[longjxchina]

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.
test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

i think isn’t the reason.
that's my code:

env value:1100
```
private static void SetThreads()
{
int miniThreads = Convert.ToInt32(Environment.GetEnvironmentVariable("MIN_THREADS") ?? "1000");
ThreadPool.SetMinThreads(miniThreads, miniThreads);
ThreadPool.GetMinThreads(out int workerThreads, out int completionPortThreads);

Console.WriteLine($"env set threads:{miniThreads}");
Console.WriteLine($"worker threads:{workerThreads}");
Console.WriteLine($"completionPort threads:{completionPortThreads}");

}
````
result: (only 1 cpu)

env set threads:1100
worker threads:1
completionPort threads:1

[longjxchina]

asp.net core 3.1 on ubuntu 16.0.4:
if SetMinThreas(1000,1000), is ok. but if more than 1000, timeout happed again.
test script:
ab -n 10000 -c 1000 http://192.168.0.22:88/

https://stackexchange.github.io/StackExchange.Redis/ThreadTheft

i think isn’t the reason.
that's my code:

env value:1100

private static void SetThreads()
{
    int miniThreads = Convert.ToInt32(Environment.GetEnvironmentVariable("MIN_THREADS") ?? "1000");
    ThreadPool.SetMinThreads(miniThreads, miniThreads);
    ThreadPool.GetMinThreads(out int workerThreads, out int completionPortThreads);

    Console.WriteLine($"env set threads:{miniThreads}");
    Console.WriteLine($"worker threads:{workerThreads}");
    Console.WriteLine($"completionPort threads:{completionPortThreads}");
}

result: (only 1 cpu)

env set threads:1100
worker threads:1
completionPort threads:1

i had found the answer: https://github.com/dotnet/aspnetcore/issues/17090#issuecomment-554032855

[Greatsamps]

Just to chime in here.

Same issue as above, also running in docker. I this instance the client and redis server are on same physical machine! physical machine cpu circa 4%

StackExchange.Redis.RedisTimeoutException: Timeout performing RPOP (5000ms), next: RPOP TradeReplicatorMessage, inst: 0, qu: 0, qs: 1, aw: False, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, out-pipe: 0, serverEndpoint: 172.16.6.20:6379, mc: 1/1/0, mgr: 10 of 10 available, clientName: ln-tcmp2, IOCP: (Busy=0,Free=1000,Min=24,Max=1000), WORKER: (Busy=6,Free=32761,Min=24,Max=32767), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in /_/src/StackExchange.Redis/RedisBase.cs:line 54
at StackExchange.Redis.RedisDatabase.ListRightPop(RedisKey key, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 998
at Inovine.RedisMessagingBus.Queuer.<>c__DisplayClass13_0.b__1(ChannelMessage subMessage)

[Xiaobaicoder-pixel]

One or more errors occurred. (Timeout performing HGETALL (5000ms), inst: 0, qu: 0, qs: 4, aw: False, rs: ComputeResult, ws: Idle, in: 0, in-pipe: 127930169, out-pipe: 0, serverEndpoint: 192.168.1.61:6379, mgr: 8 of 10 available, clientName: WIN-2NDF9JN0529, IOCP: (Busy=1,Free=999,Min=4,Max=1000), WORKER: (Busy=1,Free=32766,Min=4,Max=32767), v: 2.0.600.65315 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts))
I Use .Net Core 3.1 ,also meet this problem!!!