Rq: How about consider use Redis stream structure to implement MQ

[cw1427]

It is not a issue, I just want to discuss about Redis stream consumer group to implement MQ would be more easier for the tasks queue implement. How about your opinion?

[selwin]

I agree with you. I think Redis stream would be perfect for RQ. This is something we can consider for the future, probably a few years in when more systems ships with Redis 5.

[cw1427]

@selwin : Do we have plan to implement it? Cause Redis 6 GA is published, I'm so eager to have Steam feature to make RQ be more reliable and flexible.

[selwin]

@cw1427 I think it's still a bit early to move to Redis streams because most server distros still ship with Redis 4.X by default.

You mentioned that adopting Streams will make RQ more reliable and flexible. I'm just curious as to what features you want to see in RQ and whether we can implement them without switching to Redis Streams.

As for myself, the most prominent benefit I see in switching to Streams would be the capability to XACK jobs so they're never dropped in cases where hard failures happen after popping the job from the queue.

[cw1427]

@selwin Yes the ack would be the most charming feature to make it be more reliable, and also the consumer group feature that Redis Stream has to make it flexible to deal with mulity kinds messages generation by different applications like "my Gerrit hook events, my Jenkins jobs trigger event, my JIRA actions and so on".

I would prefer RQ could be the general "pipe" like event bus to keep all of messages coming like steam and be consumed by customer clients by goups.

Here I fould a project shipped to Redis stream already. https://github.com/robinjoseph08/redisqueue But it is Go stack. I'm not good at customizing it than RQ.