Stackexchange.redis: Sentinel support

[mcolussi]

Just a reminder to port the Sentinel code from Booksleeve to StackExchange.Redis.

ConfigurationOptions allows you to specify the Sentinel ServiceName, but connecting to a Sentinel doesn't work:

SENTINEL_IP:26379,serviceName=mymaster,keepAlive=5,syncTimeout=1000,allowAdmin=True,connectTimeout=5000,abortConnect=False
1 unique nodes specified
Requesting tie-break from SENTINEL_IP:26379 > __Booksleeve_TieBreak...
Allowing endpoints 00:00:05 to respond...
SENTINEL_IP:26379 faulted: ProtocolFailure on PING
SENTINEL_IP:26379 failed to nominate (Faulted)
> UnableToResolvePhysicalConnection on GET
No masters detected
SENTINEL_IP:26379: Sentinel v2.8.4, master; keep-alive: 00:00:05; int: Disconnected; sub: Connecting; not in use: DidNotRespond
SENTINEL_IP:26379: int ops=5, qu=0, qs=0, qc=0, wr=0, sync=5, socks=1; sub ops=2, qu=0, qs=0, qc=0, wr=0, sync=2, socks=2
Circular op-count snapshot; int: 0+5=5 (0.50 ops/s; spans 10s); sub: 0+2=2 (0.20 ops/s; spans 10s)
Sync timeouts: 0; fire and forget: 0; last heartbeat: -1s ago
Starting heartbeat...
ConfigurationChanged endpoint: SENTINEL_IP:26379
ConnectionFailed endpoint: SENTINEL_IP:26379, exception: ProtocolFailure on SENTINEL_IP:26379/Interactive, input-buffer: 663, outstanding: 0, last-read: 0s ago, last-write: 0s ago, keep-alive: 5
s, pending: 0, state: ConnectedEstablishing, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago

[bapti]

I'm looking to integrate this library and we're running a cluster of 4 redis 2.8 servers with sentinel in front of them. Should I wait until this is fixed before using this library?

[mgravell]

at the moment, you would need to do the sentinel resolution yourself; I
would probably recommend holding off (or alternatively: hack it in and
submit a PR!).

On 25 April 2014 11:01, bapti [email protected] wrote:

I'm looking to integrate this library and we're running a cluster of 4
redis 2.8 servers with sentinel in front of them. Should I wait until this
is fixed before using this library?

—
Reply to this email directly or view it on GitHubhttps://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-41377006
.

Regards,

Marc

[bapti]

Why not, I'll give it a bash over the weekend! Will I mainly be doing work in the ConnectionMultiplexer?

[mgravell]

Yes, it is basically all ReconfigureAsync. I would probably start by
special-casing the scenario where there is a ServiceName, and issuing a
"SENTINEL get-master-addr-by-name {ServiceName}" to all all endpoints in
the configuration data, then await the FIRST (of any) response from that
(with timeout), and continuing. Note that it is important that connections
that are known to be sentinels need to have their ServerType changed, so
that they are not considered as database targets.

Don't worry if you can't get to it - it is very much on my roadmap; but I
am presenting at a user-group on Tuesday, and need some time to finish prep
for that before getting stuck into SE.Redis changes

Marc

On 25 April 2014 14:22, bapti [email protected] wrote:

Why not, I'll give it a bash over the weekend! Will I mainly be doing work
in the ConnectionMultiplexer?

—
Reply to this email directly or view it on GitHubhttps://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-41391527
.

Regards,

Marc

[bapti]

Thanks Marc, just a heads up if you're testing sentinel. I've logged an issue because the nuget binaries throw errors when running in sentinel mode. They work fine if you get the binaries direct from the 2.8.4 branch.

https://github.com/MSOpenTech/redis/issues/102

[ostap36]

Hello @mgravell any chance this has been implemented and released or is upcoming? I'm seeing some of the sentinel plumbing in the code but it doesn't look final. Any updates on this are much appreciated. Thank you.

[FlagSystemes]

+1

[rstrazz]

I would pay for a 100% high end sentinel implementation that I could use in production with no worries.

[mgravell]

important see the question at the end.

OK, OK. I hear y'all. Folks want improved sentinel support. I will bump
this up and start work on this ASAP.

Apologies for delay - simply: I have limited resources (mainly: time) and
multiple projects to support. Part of the reason this has been lower
priority _for me personally_ is that we simply don't use sentinel
internally, so it is hasn't been on my critical path. This will be the next
major item I work on for SE.Redis.

As a non-sentinel user (or is that a sentinel non-user?), my immediate
priority is service resolution during connect / reconnect. So, question:

What, other than automatic service resolution, are the core pieces needed
for a useful level of sentinel provision?
On 27 Sep 2014 05:19, "rstrazz" [email protected] wrote:

I would pay for a 100% high end sentinel implementation that I could use
in production with no worries.

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-57041886
.

[rstrazz]

http://redis.io/topics/sentinel-clients

As long as everything on this page is implemented we should be good!

[ostap36]

@mgravell one core piece is the driver being tolerant of sentinel's fail over. It's implicit from the sentinel-client link rstrazz mentioned above, but I just wanted to highlight this. For the time interval that sentinels are acknowledging and performing a failover with redis instances not being accessible, driver needs to make sure that it's handled gracefully (e.g. not starving thread pool, not locking with synchronous calls...) Thank you.

[pkpjpm]

Just to follow-up on what @ostap36 said - we've been using Booksleeve in production and we haven't had a problem with sentinel negotiation per se. However we have had a problem with blocking calls to the connection utility while sentinel negotiation is underway. What we want is to avoid blocking while the failover is in operation, but to retry the connection without undue delay so that we can minimize the cache downtime during failover. This code has proven a little trickier than we would have liked, because we have blocking of service calls on one side, and thread starvation from too many retries on the other. Is it asking too much to have reactive or event based retry logic so that the client code doesn't have to deal with this? Of course, now that I'm asking this, I realize we could just send you a PR!

Looking forward to seeing how this issue evolves.

[ajwharton]

I am also looking for a client impl that can handle the masterless period when the sentinels are negotiating a new master. In testing, I just get an exception thrown which I can handle internally, but would rather see the client do something with it (option to block/wait?). We are currently configured to have very little acceptable downtime (5 sec) before the sentinels elect a new master.

[johnkwaters]

What is the status of adding sentinel support? Should I stay tuned? Any ETA?

[mgravell]

Fair question. I don't have a pressing business need for sentinel support,
so this comes out of my personal time. My personal time is evenly divided
between redis and capnproto at the moment, but my redis time is geared
towards preparing for my NDC redis talk in early December. As such, I can't
see it happening before then. It all depends on how much time I can scrape
together!
On 18 Nov 2014 21:03, "John Waters" [email protected] wrote:

What is the status of adding sentinel support? Should I stay tuned? Any
ETA?

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-63544196
.

[johnkwaters]

Thanks for the quick response! The background is I am using redis both as a queing system, as a SignalR backplane, and as a cache for Telerik reporting. This is in a web farm, but now Redis has become a single point of failure. With StackExchange.Redis sentinel support, if it hides behind the ConnectionMultiplexer, then I can keep using this setup, but otherwise I need to look for other backplane solutions where the backplane is HA. So I was just curious as to whether this is a likely scenario or not.
Cheers
John

[rstrazz]

I also need a way to guarantee that Redis won't be a single point of failure too. I want to spin up two Redis servers, plug both of their IP:port strings into ASP.NET, and have little to no downtime if one of the servers fail. So basically an implementation of this:

http://redis.io/topics/sentinel-clients

The part at the bottom "Subscribe to Sentinel events to improve responsiveness" is really just a nice to have. By the way, capnproto looks awesome!! Thanks for your time :)

[rstrazz]

Marc,

Would the Sentinel implementation in this project give you a kick start at all?

https://github.com/ctstone/csredis

Thanks!

[mgravell]

I have the starts of a sentinel implementation I can steal from booksleeve

• although IIRC there have been a good few sentinel tweaks in the interim.
  The problem is simply a human factor: me - and the lack of infinite time.
  On 4 Dec 2014 19:00, "rstrazz" [email protected] wrote:

Marc,

Would the Sentinel implementation in this project give you a kick start at
all?

https://github.com/ctstone/csredis

Thanks!

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-65683548
.

[rstrazz]

OK no worries for now man, I wouldn't need this until Jan/Feb. :) If I need it sooner I'll post back on here and perhaps I could pay you for it.

[mgravell]

While the sentiment is noted, I'm not sure that's an agreeable way to
communicate it... Let's play nicely, folks.
On 10 Dec 2014 21:28, "Jake Scott" [email protected] wrote:

@rstrazz https://github.com/rstrazz Why don't you implement this
yourself and stop being so f'n pushy!!

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-66527081
.

[Buckers81]

mgravell - if you don't use sentinel, what is your strategy for handling redis failures?

[mgravell]

We have master/slave/slave/slave for limited recovery. At the moment actual
failover is largely manual and human. We're looking to make this smarter
when "cluster" is viable for us.

Does that help?
On 26 Jan 2015 21:05, "Buckers81" [email protected] wrote:

mgravell - if you don't use sentinel, what is your strategy for handling
redis failures?

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-71538027
.

[Buckers81]

Interesting. We've found sentinel and automatic recovery a right PITA. Also looking forward to cluster if it helps with this.

[CamW]

Had a stab at this since we needed it and submitted a PR here https://github.com/StackExchange/StackExchange.Redis/pull/121 if you already have a plan for an implementation from booksleeve this probably won't be in line with that but let me know what you think or if I can help.

[sdebnath]

@mgravell bump

[cjb5790]

@mgravell Any update on this?

[mylesjj]

@mgravell Any news on this?

[sreefubs]

@mgravell Is Sentinel support there in StackExchange.Redis now ?

[chester89]

@sreefubs I believe there's a comment on some issue saying "it works in
most cases" or something like that

2016-01-08 10:28 GMT+03:00 sreefubs [email protected]:

@mgravell https://github.com/mgravell Is Sentinel support there in
StackExchange.Redis now ?

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-169917459
.

Yours faithfully,
Gleb

[sreefubs]

@chester89 I’m new to Redis and we’re analyzing the tool for a specific requirement which needs High-Availability & Consistency.

Environment Details:
Redis Server(Windows OS) : v2.8.2104 https://github.com/MSOpenTech/redis
.Net SDK : https://github.com/StackExchange/StackExchange.Redis
Cluster Setup : 1 Master & 3 Slaves & 3 Sentinels.

Test Scenario :
While iterating Redis SET command from .Net client thro StackExchange.Redis SDK, I'm crashing redis-master node.

Once Redis Sentinel elects new master (one of the slaves), now how will my application get updated about the new redis master IP?

[chester89]

If I'm not mistaken, there's a PR that addresses your problem:
https://github.com/StackExchange/StackExchange.Redis/pull/121
It's out of date, I believe, but the code does just that - it sets up a
cluster, shuts down current master and waits to see if the switch occurs

2016-01-08 15:16 GMT+03:00 sreefubs [email protected]:

@chester89 https://github.com/chester89 I’m new to Redis and we’re
analyzing the tool for a specific requirement which needs High-Availability
& Consistency.

Environment Details:
Redis Server(Windows OS) : v2.8.2104 https://github.com/MSOpenTech/redis
.Net SDK : https://github.com/StackExchange/StackExchange.Redis
Cluster Setup : 1 Master & 3 Slaves & 3 Sentinels.

Test Scenario :
While iterating Redis SET command from .Net client thro
StackExchange.Redis SDK, I'm crashing redis-master node.

Once Redis Sentinel elects new master (one of the slaves), now how will my
application get updated about the new redis master IP?

—
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-169983278
.

Yours faithfully,
Gleb

[aishelm]

Marc, I have a question related to this topic: should the sentinels be registered as hosts along with the redis servers in the configuration?
If the answer is NO, would it be different in the future when we improve coverage for sentinel beyond what currently exists?

[aaronmell]

@mgravell @NickCraver So the company I work for is paying me to implement sentinel support. Its a feature we really want, we don't want to switch clients to get it, and we can't wait for the feature to be added by someone else. I can follow the thread here and the current thoughts on implementation. Any other guidance or information you could provide that could help make this successful would be appreciated as I would prefer not to maintain a fork of the project long term.

[ajwharton]

@aaronmell IIRC sentinel support is in the Java SDK. If so, starting there and doing a port would be a thing.

[mgravell]

Since I don't use sentinel routinely, I simply don't know enough about it
_as it currently exists_ to define what it needs to do. My naive assumption
is that _simple_ usage should just list a sentinel node or nodes in the
config string, and a service name, and the connect API should just figure
everything out. For _that_ it should just need some hacks (big or small) to
the connect code - and as long as the different connections are correctly
marked, it _should_ all work.

As for what _additional_ APIs should be used: don't know!

Any use?

Marc

On 29 March 2016 at 16:18, Aaron Mell [email protected] wrote:

@mgravell https://github.com/mgravell @NickCraver
https://github.com/NickCraver So the company I work for is paying me to
implement sentinel support. Its a feature we really want, we don't want to
switch clients to get it, and we can't wait for the feature to be added by
someone else. I can follow the thread here and the current thoughts on
implementation. Any other guidance or information you could provide that
could help make this successful would be appreciated as I would prefer not
to maintain a fork of the project long term.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-202949046

Regards,

Marc

[aaronmell]

@mgravell That helps. I dug into the code in the connection multiplexer and I think I have an idea about how to approach it.

The multiplexer just needs some code to add the master sentinel is monitoring.
Need to add some logic somewhere (not sure where yet) to trigger a reload of configuration when sentinel publishes a message that a master has been promoted.

I don't know if that is everything the client needs, but that covers the general use case I think.

[Areson]

Any movement on this? I'm also in need of a Sentinel implementation. I've rolled my own changes for it into the ConnectionMultiplexer, but I wanted to hold on pushing it out if folks already had something that was forthcoming.

[aaronmell]

Sorry, i ended up going the same route
On May 17, 2016 1:18 PM, "Ian Oberst" [email protected] wrote:

Any movement on this? I'm also in need of a Sentinel implementation. I've
rolled my own changes for it into the ConnectionMultiplexer, but I wanted
to hold on pushing it out if folks already had something that was
forthcoming.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/StackExchange/StackExchange.Redis/issues/22#issuecomment-219806985

[Areson]

Quick design question since I'm on the subject at the moment: Is there an overwhelming preference on how should be handled? I noticed that PR #121 handled it by having two ConnectionMultiplexers, one for the Sentinel connection, which was passed off to a second ConnectionMultiplexer as a configuration option. I riffed on that a bit but preferred not to directly handle two connections, instead having the ConnectionMultiplexer recognize that it was connecting to Sentinel(s) and internally create a second, managed, ConnectionMultiplexer that could be accessed via a property, ala:

``` C#
ConfigurationOptions sentinelConfig = new ConfigurationOptions();
sentinelConfig.ServiceName = "MyServiceName";
sentinelConfig.EndPoints.Add("127.0.0.1", 26379);
sentinelConfig.EndPoints.Add("127.0.0.2", 26379);
sentinelConfig.TieBreaker = "";
sentinelConfig.CommandMap = CommandMap.Sentinel;

// Need Version 3.0 for the INFO command?
sentinelConfig.DefaultVersion = new Version(3, 0);

// Create config options for connection to the master
ConfigurationOptions masterConfig = new ConfigurationOptions();

sentinelConfig.SentinelMasterConfigurationOptions = masterConfig

// Connect!
ConnectionMultiplexer sentinelConnection = ConnectionMultiplexer.Connect(sentinelConfig);
ConnectionMultiplexer masterConnection = sentinelConnection.SentinelMasterConnection;

Another route would be to recognize that we are connecting to Sentinel(s) and internally create the connection to the master and pass that back instead, sidestepping having two ConnectionMultiplexer objects exposed to the end user:

``` C#
// Same setup as before goes here

// Connect!
// We directly get back the ConnectionMultiplexer for the masters and the Sentinel connection
// is internally managed for us
ConnectionMultiplexer masterConnection = ConnectionMultiplexer.Connect(sentinelConfig);

Thoughts and/or preferences?

[Areson]

Also, see PR #406 for more details.

[CamW]

@Areson I may be wrong but it looks like you're considering it a foregone conclusion that someone using the client to connect to a sentinel also wants to connect to the dictionary server with that sentinel's advice, this isn't necessarily the case. One may just want to interact with the sentinel alone. Perhaps consider a method or config option to indicate that the master server advised by the sentinel server should be connected to.
On a separate note, I'm pretty bleak that @mgravell has been so unresponsive on this. My PR, #121 has been open for 18 months and is pretty stale at this point and I would have really liked to contribute a solution. In that time I haven't been able to get even a single comment on the matter out of the powers that be.

[Areson]

@CamW I'd say that's a fair assessment. I'm a fairly recent user of Redis, so I have a pretty narrow use case that I am working with. It would be easy enough to have it forgo connecting to the master. What about something like this:

``` C#
ConfigurationOptions sentinelConfig = new ConfigurationOptions();
// Don't specify a ServiceName when connecting to the Sentinel, as it can have
// several masters that it monitors
//sentinelConfig.ServiceName = "MyServiceName";
sentinelConfig.EndPoints.Add("127.0.0.1", 26379);
sentinelConfig.EndPoints.Add("127.0.0.2", 26379);
sentinelConfig.TieBreaker = "";
sentinelConfig.CommandMap = CommandMap.Sentinel;

ConnectionMultiplexer sentinelConnection = ConnectionMultiplexer.Connect(sentinelConfig);

// Get a managed connection to a master when I want it
ConnectionOptions masterConfig = new ConfigurationOptions();
// Specify the service name of the master I want to connect to
masterConfig.ServiceName = "MyServiceName";
ConnectionMultiplexer masterConnection = ConnectionMultiplexer.Connect(masterConfig, sentinelConnection);
```

Though I suppose at this point it could simply take the same approach your PR did and pass the Sentinel connection as a config option. I did nab a few bits of code from your PR (thanks!) so switching it back to that approach would be pretty straightforward.

On the other note, @mgravell has been committing PRs very recently, so I don't think that we are dead in the water. It may simply be that he felt like the effort had stalled and wasn't contributing any attention to it. We'll have to see if and what he says. I know my perception based on this thread and #121 was that things hadn't quite gotten resolved and then died out.

[aaronmell]

The approach I took in my code, was to treat them separately. I go to sentinel and get the configuration for redis. Then I hook up to the events that a slave has failed over to a master. When that happens, I reset my Multiplexer

[Areson]

That's certainly an option. I wanted more of the leg work to be done behind of the scenes so that if I asked for a master from a Sentinel connection that what I got would be governed automatically without needing me to setup any special event handling myself. Though perhaps I'm misinterpreting the approach you mentioned?

[moxplod]

Any ideas on when this is likely to be merged?

[moxplod]

Next best thing - Does anyone has a fork of this that has sentinel support :)

[Areson]

@moxplod Yes. See #406 for the pull request from my fork.

[lexxdark]

Hey, I have just created a pull request updating the sentinel support, for your consideration. Pull request #692

[NickCraver]

Tracking this in #692 now - we were really hoping the community who's more familiar with Sentinel behavior could help us out there, but no luck thus far. It's really all we're lacking to merge Sentinel support in. Regardless - collapsing into #692 so all discussion happens on the PR.