Framework: Dynamic Queues not working

[ARKKYN]

• Laravel Version: 5.5
• PHP Version: 7.1
• Database Driver & Version: Redis

Description:

@taylorotwell I am using Laravel queues to process jobs in a connection , since i have a dynamic queue name , I cannot specify queue name when using queue:work command, since I tried search a solution for it in laravel and other forums , It turned out to be a dead end

Steps To Reproduce:

Create a dynamic queue name while scheduling queues and check it in horizon dashboard

[laurencei]

Why are you using dynamic queue names in the first place? Just use dedicated queues?

[ARKKYN]

@laurencei we have a requirement to use multiple queues. and need to process it simultaneously, since we have lot of queue connections already for many things Notifications, and few Variety of jobs in which the jobs in the queue has equal priority. in scaled environment we have time lags for processing bunch of jobs in one or predefined queues

@taylorotwell Sorry to mention again here if there is option to process queue on dynamic names it would be help full and also no option to set queue connection on Broadcast class or On event Class

[sisve]

Have you considered building a custom queue that would fetch from multiple queues? As an example, it could involve extending the DatabaseQueue and just remove the WHERE queue = $queue part of the logic that selects a new job. Or change it into WHERE queue LIKE 'my-queues-%'

[ARKKYN]

@sisve Thanks, But may i know At which Class that handles RedisQueue, If this feature is added it would be good

[sisve]

I do not know the redis commands well enough to tell you how to implement this, but the current system is based on RedisQueue::retrieveNextJob which uses a lua-script found in LuaScripts::pop.

At a first glance I do not see anyway to rewrite the script to lpop from any of the given queues. It looks like you will always have to poll redis for one queue at a time, so your best bet (in my naive eyes) is to have a list of all queues to fetch from, and shuffle it randomly. This way you'll still take the first available job you find, but the order you process the queues will differ everytime you call retrieveNextJob.

[4n70w4]

@laurencei The same problem! I need to handle batch of tasks, for example, 1 000 pieces. And I need to be able to stop the batch of tasks and see the progress of the handling. This is most conveniently done through a dynamic queue name. One unique queue name for each batch of tasks.