Azure-functions-host: Queue Trigger Node template does not handle message queued from Python script

[Revocsid]

I followed this tutorial to add some message in my queue from a Python script : https://azure.microsoft.com/en-us/documentation/articles/storage-python-how-to-use-queue-storage/
Messages were correctly added since I can retrieve them with my Python script by using the get_messages function.

However I've created an Azure function with the Queue Trigger Node template to handle these messages added by my Python script but my function isn't called when a message is queued.
I've contacted the azure support which after some tests advised me to open an issue on this repo.

[mathewc]

Can you point us to a repo containing a repro of this so we can investigate? The Azure Function Node queue trigger is just listening on the queue and really doesn't care how the messages get in there - e.g. via whatever client you have, Storage Explorer, etc. If you can help us repro your specific scenario we can get to the bottom of it, thanks :)

[Revocsid]

Hi,
You can find a simple script in this repo : https://github.com/Revocsid/azure-webjobs-issue-434
I just followed the tutorial that I mentioned in my previous message. By the way I noticed a mistake in this tutorial. Indeed to delete a message the tutorial mentioned to use :

messages = queue_service.get_messages('taskqueue') for message in messages: print(message.content) queue_service.delete_message('taskqueue', message.message_id, message.pop_receipt)

But with the latest version of the azure-storage package, message_id doesn't exist and you have to use the "id" property :

messages = queue_service.get_messages('taskqueue') for message in messages: print(message.content) queue_service.delete_message('taskqueue', message.id, message.pop_receipt)

Thanks for your help !

[christopheranderson]

Do you have the azure storage package uploaded? Your sample doesn't include any information about adding the package.

You should be able to run your run.py locally to test and confirm. Nothing you're doing in your code/setup should prevent the function from running unless it is non-runnable code (syntax errors, missing packages, etc.).

This might be a usability issue related to how we express runtime errors from "Script" types.

[Revocsid]

Hi Christophe,

Yes I'm using the azure storage package ( https://github.com/Azure/azure-storage-python ) which is recommended by the microsoft tutorial.

My code is running well, I can push messages in the queue and even retrieve them.
My problem concerns my Azure function (Queue Trigger Node template) which is not called when a message is pushed in the queue.

[mathewc]

Can you point us to a repo containing a repro of this?

[Benjiiim]

Hi Mathew,
A repro repo has been sent earlier in the thread: https://github.com/Revocsid/azure-webjobs-issue-434
Isn't it enough?
Thanks!

[mathewc]

Ah, sorry I missed that, thanks.

[brettsam]

It looks to be a message encoding issue with the Azure Python SDK and unrelated to functions. When I'd run the python script in the sample above (after tweaking a few things b/c I was using Python 3), it would throw an error in the Azure Storage Explorer when I tried to look at the message. Same thing with trying to look at it with the VS Cloud Explorer. The error was The input is not a valid Base-64 string as it contains non-base 64 characters, more than two padding characters, or an illegal character among the padding characters.

Looking through the issues linked from the documentation link above, I found: https://github.com/Azure/azure-storage-python/issues/18

I changed the script as follows:

1. import QueueMessageFormat: from azure.storage.queue import (QueueService, QueueMessageFormat)
2. after creating the queue_service, set its encode_function -- you could also set the decode_function: self.queue_service.encode_function = QueueMessageFormat.text_base64encode
3. remove the unicode conversion and just create the message as: message = json.dumps(obj)

And with those changes, the messages appeared correctly in the queue and my function handled them.

@Revocsid -- could you give these steps a try and see if it gets things working?

I'm not sure why our logs didn't surface the error, that would have been helpful. I'm going to see if this is something we can catch.

[brettsam]

We should be able to handle this error better, but that will be done in the core SDK repo. I've created issue https://github.com/Azure/azure-webjobs-sdk/issues/768 to track this.

I'll consider this issue closed unless @Revocsid comes back to us with a problem and re-opens it.

[jpoon]

I was hitting the same issue as @Revocsid. Namely pumping data in using azure-storage-python SDK. @brettsam's recommendation worked:

        from azure.storage.queue import QueueService, QueueMessageFormat

        queue = QueueService(..)
        queue.create_queue(..)
        queue.encode_function = QueueMessageFormat.text_base64encode

        queue.put_message(..., unicode(message))

Still needed to convert the message to unicode, as the put_message does validation for that: https://github.com/Azure/azure-storage-python/blob/61f686104bf86211a85662dd1a28329482ae9f75/azure/storage/queue/models.py#L103, but at least now WebJob is :ok_hand: and doesn't freak out when handling the message.

[4c74356b41]

I can confirm I was hitting this issue, but the code above solved it. As a free tip, you might want to post it in some reasonable place in azure functions docs or on the storage queue python examples...

[giantryansaul]

Considering this is a fairly old discussion and I just ran into the same problem myself while writing a very simple script, it seems like this should be included in the standard documentation: https://docs.microsoft.com/en-us/azure/storage/queues/storage-python-how-to-use-queue-storage