Resque: Does Resque support a Redis Cluster?

[rulloa]

We are implementing a Redis cluster, but we want to know if Resque can work with A Cluster or not.

Regards

[JFarber]

+1

[sameg14]

+1

[krobi64]

There's been no answer on this yet?

[steveklabnik]

@krobi64 yes, I've been a bad maintainer, I apologize.

My understanding of Redis Cluster is that it automatically handles the cluster aspects for you, so it should Just Work. I have not tried it myself.

If someone has, a report would be valuable.

[roychri]

@steveklabnik The Redis cluster does not support multiple keys operations, transaction or lua scripts. Well, it kinda support it if you use the same hash values in keys you are using (so they are all coming from the same node). But Resque is not using any hash at all at the moment.

So the real question should be... Is Resque doing any type of multiple keys operations, transactions or lua scripts?

Examples of multiple keys operations are BITOP, BLPOP, BRPOP, MGET, MSET, PFMERGE, PFCOUNT, SDIFF, SDIFFSTORE, SUNION, SUNIONSTORE, SINTER, SINTERSTORE, ZINTERSTORE, ZUNIONSTORE, EVALSHA,
Or something like "DEL key1 key2" or "EXISTS key1 key2"

Resque does not seem to be using transactions or lua scripts, but...

A quick scan and I found mget being used:

https://github.com/resque/resque/blob/v1.25.2/lib/resque/worker.rb#L79

This would not be supported by the Redis Cluster since those multiple keys would most likely be coming from different nodes. If this is changed to perform individual gets, it _should_ work.

[ramonsnir]

Don't know if this is the only (living) issue on this topic, but: NO Resque does not support Redis Cluster (just tried locally :) ). It is running at least one transaction, somewhere (MULTI/EXEC).

This means that it needs to modify its keyspace, so all keys will sit in the same slot - or at least those of the multi-key operations.

Up to you, @steveklabnik , to prioritize but you should still know.

[brettrann]

@steveklabnik should this ticket be open? From what Ramonsnir said it doesn't work with Redis Cluster.

[januszm]

@steveklabnik yes it does not work with Redis in Cluster mode. If you try, you'll see log messages like:

NameError: uninitialized constant Redis::Distributed

but then even if you try require 'redis/distributed'
you'll end up with:

> Resque.info
Redis::CommandError: CROSSSLOT Keys in request don't hash to the same slot

It's worth adding this information to the Readme

[jbmyid]

Resque dont have redis cluster mode support yet?
Getting following error while using in cluster mode.

Redis::Cluster::CrossSlotPipeliningError: Cluster client couldn't send pipelining to single node. The commands include cross slot keys. ["resque:queues", "resque:queue:training"]

[alejandromarcu]

@jbmyid , I was looking for whether Resque works in clustered mode and found this thread.
I haven't tried it yet, but if you're having that problem, maybe it can be solved using https://redis.io/topics/cluster-spec#keys-hash-tags
For example, setting the redis namespace to {resque} so that all the keys are in the same shard

[dhirajkasar]

@jbmyid did you get it working ? I am getting same error when redis is setup in cluster mode.

Redis::Cluster::CrossSlotPipeliningError (Cluster client couldn't send pipelining to single node. The commands include cross slot keys. ["resque:queues", "resque:queue:test_queue"])

[laranicolas]

@jbmyid any news on this? having the same issue.

[jbmyid]

@alejandromarcu @dhirajkasar @laranicolas
No, did'nt get anything. Was working on some POC.