StackExchange.Redis timeout and “No connection is available to service this operation”

[stilago-ci]

Hi Marc,

First of all thanks for this awesome redis client.
But I have some issues :)

I have the following issues in our production environment (Web-Farm - 4 nodes, on top of it Load balancer):

1) Timeout performing HGET key, inst: 3, queue: 29, qu=0, qs=29, qc=0, wr=0/0
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in ConnectionMultiplexer.cs:line 1699 This happens 3-10 times in a minute

2) No connection is available to service this operation: HGET key at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT in ConnectionMultiplexer.cs:line 1666

I tried to implement as Marc suggested (Maybe I interpreted it incorrectly) - better to have fewer connections to Redis than multiple. I made the following implementation:

public class SeRedisConnection
{
private static ConnectionMultiplexer _redis;

private static readonly object SyncLock = new object();

public static IDatabase GetDatabase()
{
    if (_redis == null || !_redis.IsConnected || !_redis.GetDatabase().IsConnected(default(RedisKey)))
    {
        lock (SyncLock)
        {
            try
            {
                var configurationOptions = new ConfigurationOptions
                {
                    AbortOnConnectFail = false
                };
                configurationOptions.EndPoints.Add(new DnsEndPoint(ConfigurationHelper.CacheServerHost,
                    ConfigurationHelper.CacheServerHostPort));

                _redis = ConnectionMultiplexer.Connect(configurationOptions);
            }
            catch (Exception ex)
            {
               IoC.Container.Resolve<IErrorLog>().Error(ex);
                return null;
            }
        }
    }
    return _redis.GetDatabase();
}

public static void Dispose()
{
    _redis.Dispose();
}

}

Actually dispose is not being used right now. Also I have some specifics of the implementation which might cause such behavior (I'm only using hashes):

1. Add, Remove hashes - async
2. Get -sync

Original question you can find here:
http://stackoverflow.com/questions/22930856/stackexchange-redis-timeout-and-no-connection-is-available-to-service-this-oper

Thanks a lot in advance!

Looking forward for your response.

BTW: Here what I can see from my redis cache server (TCP connections never goes up):

Also I have stress-testing web-farm (2 nodes - Microsoft NLB). If I run load test for 200 req/s - it handles it ok in this environment and I noticed 5-8 open TCP connections and no timeouts at all. But the issue is with Microsoft NLB if you are making stress-test from 2-3 ip addresses it allocates requests only to one node.

So in production environment case it should be 4 nodes * 8 (active connections) = 32 open connections rather than 200.

[stilago-ci]

Hi Marc,

Actually I found your answer regarding connection timeout - and it helped me a lot!
Thanks the issue has been solved.

[mgravell]

K; sorry for delay, I have been away from a computer

[anisimovyuriy]

Hi Marc,

Thanks a lot for response - of course I understood :)

[anisimovyuriy]

But still I have the same issue with timeouts and it is reproduced only in the following circumstances:
under 1.5 - 2.5GB volume in cache - no exceptions, after it is just polluting our Elmah DB with timeouts.
Also I asked for investigation our IT dep. regarding network connectivity which appeared - it's not an issue.

Right now I'm setting up MSOpen redis 2.8 - I hope it helps. Currently it is 2.4.6.

[mgravell]

I'm still unclear about how I would repro this...

On 14 April 2014 13:29, Yuriy Anisimov [email protected] wrote:

But still I have the same issue with timeouts and it is reproduced only in
the following circumstances:
under 1.5 - 2.5GB volume in cache - no exceptions, after it is just
polluting our Elmah DB with timeouts.
Also I asked for investigation our IT dep. regarding network connectivity
which appeared - it's not an issue.

Right now I'm setting up MSOpen redis 2.8 - I hope it helps. Currently it
is 2.4.6.

—
Reply to this email directly or view it on GitHubhttps://github.com/StackExchange/StackExchange.Redis/issues/12#issuecomment-40360143
.

Regards,

Marc

[anisimovyuriy]

Hi Marc,

Actually I found initial issue and there is nothing to do with StackExchange.Redis client as it is working great! Here is what I came up with in order to solve this issue:
http://stackoverflow.com/questions/22930856/stackexchange-redis-timeout-and-no-connection-is-available-to-service-this-oper

See "Update 2".

Thanks a lot for such nice product!

[minjie-kooboo]

I always meet the same problem too.

1. Timeout exception
2. No connection is available to service this operation

[zhangpengchen]

Me too :(
We are using azure redis sometimes we got timeout exception.
Timeout performing GET Setting:SettingName=Emails.ConfirmationEmail.Url, inst: 1, queue: 2, qu=0, qs=2, qc=0, wr=0/0, in=181/0rnStackTracern at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImplT

[anisimovyuriy]

Hi guys there is nothing to do with Redis.StackExchange - it's a nature of Redis!
When we are talking about the context of read/write without memory limits - Redis the best, but when Redis tries to release some memory - there is always a challenge as Redis single-threaded solution. While you have a lot of read/write operations and if Redis tries to reclaim some memory this one thread is blocked to reclaim the memory - of course it depends how much memory your environment uses and amount of read/write operations. I had struggled 2 months with analysis as well as our company had engaged 2 other third party company to help us - unfortunately no success we mutually came to the same point. Then we moved to AppFabric - there were a lot of tweaks of course - it wasn't so easy as with Redis - but it works stable on our production environment without any flaws.
Of course there are a lot of home-made solutions to this issue like:
1) Increasing amount of nodes in the cluster to satisfy SW requrements
2) It's a good practice to install almost the same amout of instances as number of cores on one machine
3) Create home-made nodes' rotation solution

[mikkark]

I had this same issue and fortunately I managed to fix it just by increasing the connection timeout to 10 seconds. The connection to Azure Redis worked fine from my dev machine and using the Redsmin client, but our app in Azure was unable to get connected.

[MosheL]

I am using 3 servers. two of them in the same LAN with the Redis and the last one in my DR site (1-2ms).
I am seeing this error from time to time on the DR site, but not on the production.

[spuder]

@mikkark Which setting exactly did you adjust? Was it timeout 0 in the redis.conf?

[coolravi142]

i am getting the error message of "No connection is available to service this operation". Can someone explain why this error actually comes

[MosheL]

@coolravi142 check your network between IIS & redis server.

[coolravi142]

@MosheL i have checked the connection and it is connected. I have checked it through redis.IsConnected. I am getting this error whenever i am trying to execute a batch

[MosheL]

In my production LAN I can see this error after disconnect, when the connection is restored but SignalR still think that it disconnected.

[coolravi142]

Any idea of how to resolve it??

[apopovic]

I have the same issue like "coolravi142" would appreciate help for resolving errormessage: "No connection is available to service this operation".

[usernamedd]

I am facing the same issue , do not know why this happened ,also do not konw how to resolve it

[mgravell]

@usernamedd on which version? If it is < 2.0, then I'd say "try 2." - almost all of the 2. work related to connection stability.

[usernamedd]

@usernamedd on which version? If it is < 2.0, then I'd say "try 2._" - almost all of the 2._ work related to connection stability.

Thank u so much . it is actually < 2.0 , but version 2.0 need 4.7 framework . I am thinking about it .

[mgravell]

No it doesn't. It has builds from 4.6.1 upwards.

On Tue, 18 Sep 2018, 02:06 usernamedd, [e-mail] wrote:

@usernamedd https://github.com/usernamedd on which version? If it is <
2.0, then I'd say "try 2." - almost all of the 2. work related to
connection stability.

Thank u so much . it is actually < 2.0 , but version 2.0 need 4.7
framework . I am thinking about it .

—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/12#issuecomment-422219617,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABDsAMXVxW1YNY1b8MpkGTP9B0VJW_Iks5ucEcNgaJpZM4BwiiD
.