Azure-webjobs-sdk: Support triggering a single function on multiple queues (wildcard queue name)

[DarranShepherd]

Example scenario:
A webjob that monitors all *-poison queues in a storage account, with a single generic action taken on all (e.g. write queue message to blob for inspection, email/Slack/SMS to ops team to alert of failure).

I've looked into implementing this as an extension, but rapidly ended up with 90% of the code being duplication of the core SDK due to much of the infrastructure behind QueueTriggerAttribute being marked as internal.

Would you be willing to consider a pull request that creates a MultipleQueueTriggerAttribute that accepts a regex to match against queue names, creates a listener monitoring all matching queues and adds a queueName binding in addition to the existing bindings?

C# public static void HandlePoisonQueues( [MultipleQueueTriggerAttribute(".*-poison")] string message, string queueName, [Blob("poison/{queueName}/{id}.txt")] TextWriter blob, TextWriter logger) { logger.WriteLine($"Poison message received on {queueName}") blob.Write(message); }

[CarlosSardo]

Another interesting and useful scenario:

To overcome the Azure Storage Queue scalability limits more easily.

A single queue can process approximately 2,000 messages (1KB each) per second (each AddMessage, GetMessage, and DeleteMessage count as a message here). If this is insufficient for your application, you should use multiple queues and spread the messages across them.

Defining a Regex on the queue name:
[MultipleQueueTriggerAttribute("orders[0-9]")] string message

Passsing several queue names:
[MultipleQueueTriggerAttribute("orders1", "orders2", "orders3")] string message

[ivorwan]

Another interesting and useful scenario:

To overcome the Azure Storage Queue scalability limits more easily.

A single queue can process approximately 2,000 messages (1KB each) per second (each AddMessage, GetMessage, and DeleteMessage count as a message here). If this is insufficient for your application, you should use multiple queues and spread the messages across them.

Defining a Regex on the queue name:
[MultipleQueueTriggerAttribute("orders[0-9]")] string message

Passsing several queue names:
[MultipleQueueTriggerAttribute("orders1", "orders2", "orders3")] string message

Personally, I would prefer a regex or wildcard binding. That way, in order to scale, we would simply create a new azure queue, without having to update the function

[mayoatte]

Hey folks,

Thanks for all your hard work.

Is there any progress on this request or is it being considered at all? We're running into both scenarios above (a lot of poison queues to handle in a common way & approaching queue throughput limits) and this seems like a nice solution.

Thanks

[alexgman]

I need a function to be triggered by multiple queues. Any progress here?

[cocowalla]

Has anyone from Microsoft had a look at this, and is it on the backlog?

For massive scale and better tenancy isolation, 1 queue per tenant would be ideal - but that would also require one function binding per queue too, which is far from ideal (and likely to hit some scaling limitation, which are rife throughout all Azure services)

[VictorGazzinelli]

Any updates? Has anyone found a workaround for this?

[onurleblebici]

After five years, any progress or update?
At least a workaround?

[vachillo]

Also requesting a new feature/workaround for this situation

[alexgman]

We will take your request into consideration.

On Fri, May 7, 2021 at 12:48 PM vachillo @.*> wrote:

Also requesting a new feature/workaround for this situation

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-webjobs-sdk/issues/808#issuecomment-834651652,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABAIE34URGKLX6ZAECQ67Z3TMQRYHANCNFSM4COKFDKA
.