Bull: Error: Sending custom commands in pipeline is not supported in Cluster mode when use Cluster Mode

Created on 25 Oct 2017 · 16Comments · Source: OptimalBits/bull

When i use bull in redis cluster mode.

Configuration options

const opts = {
    createClient: function(type){
       return new redis.Cluster([
            ...
        ]);
    },
    prefix: '{}'
};

Create job it's worked but when i process job with queue.process,
I encounter the following error

Error: Sending custom commands in pipeline is not supported in Cluster mode.
    at Pipeline.exec (\node_modules\ioredis\lib\pipeline.js:249:19)
    at Pipeline.pipeline.exec (\node_modules\ioredis\lib\transaction.js:34:26)
    at \node_modules\bull\lib\job.js:210:18

Apparently the pipelines are not compatible with cluster mode of redis.

Can you confirm this problem?
How to use bull with a cluster redis without encountering this problem?

PRIO 1 enhancement

Source

zetflo

👍1

All 16 comments

ok, seems like it is a limitation of ioredis at least. I did not now about it: https://github.com/luin/ioredis/issues/536
This requires some rewrite in order to make it work in cluster mode unfortunately.

manast on 25 Oct 2017

@manast we are starting to run into this. Do you know what features specifically require pipelines? Is there some subset of features we can still use without them?

ajwootto on 20 Mar 2019

@manast I'm running in the same issue when calling job.moveToFailed(err). it's possible to do something else here? it's strange for me, because I use job.moveToCompleted() without any problems. thx!

btzsoft on 22 Apr 2019

@btzsoft that is because ioredis does not support multi commands using custom lua scripts, and only moveToFailed currently uses that. In order to solve this issue the method should be rewritten to not use multi, which is a bit to work to do...

manast on 22 Apr 2019

@manast oh no, do you have some plans in future to change this? if no can you give pls some hints where to take a look, maybe I will make a fork and will changes this, bcs it's really a big blocker for us. thank you for your time.

btzsoft on 22 Apr 2019

you can check the implementation of moveToFailed, it is quite inconvenient to not being able to use multi in this case. The other alternative is to convince the authors of ioredis to fix this issue:
https://github.com/luin/ioredis/issues/536

manast on 22 Apr 2019

@manast just to better understand, why moveToFailed use multi command? and why doesn't work as moveToFinished. thx!

btzsoft on 23 Apr 2019

@btzsoft I think you could understand this better if you check the source code, basically it is because moveToFailed has much more functionality such as retries, updating attempts, backoffs, etc.

manast on 23 Apr 2019

@manast Does this mean bull is incompatible with a Redis cluster-mode-enabled cluster until ioredis is enhanced? Is there no workaround?

In the meantime, can you update the Patterns doc, which provides a (broken?) example of how to use Redis Cluster?

https://github.com/OptimalBits/bull/blob/develop/PATTERNS.md#redis-cluster

joebowbeer on 2 Jun 2020

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

akhil-karat on 11 Jul 2020

@manast Can you confirm that this is not an issue in BullMQ?

joebowbeer on 13 Aug 2020

@joebowbeer it is the same problem unfortunately. I was expecting ioredis to be fixed but for some reason they did not prioritize it. The alternative is to convert "moveToFailed" method in bull/bullMQ to one lua script, as you can see it is quite a bit of job, but it seems like it is the only way to solve this issue once and for all... will try to prioritize it.

manast on 14 Aug 2020

👀1

@akhil-karat

@manast - I am facing same problem. Works with local redis instance. But in our int and prod we use Redis cluster. When a job is added to queue process fails with error 'Sending custom commands in pipeline is not supported in Cluster mode'. Then job stalls. Is there a work around for this issue? It is really urgent for us. Kindly help.

Are you sure this happens when you add the job and not when the job fails?

manast on 14 Aug 2020

@manast Thanks for prioritizing this! I repo'd the issue in BullMQ as well.

joebowbeer on 14 Aug 2020

👀2

Super sorry for the excessive mentions all! (Was doing a bit of cleanup and of course GH has appended to this discussion for every instance of these commits.)

That said, if anyone urgently needs this working, a patch can be found in the fpm-git/bull/bug/fix-#758-redis-cluster-breakage branch.

Insert the usual "it works for us, but no warranties, etc." spiel here.