Azure-sdk-for-net: [FEATURE REQ] Base64 Encoding helpers in QueueClient.SendMessage

[svengeance]

Library or service name.
Azure.Storage.Queue

Is your feature request related to a problem? Please describe.
In the previous version of the queue SDK, messages were automatically base64 encoded whether you wanted them to be or not - https://github.com/Azure/azure-storage-net/blob/master/Lib/Common/Queue/CloudQueueMessage.Common.cs#L267

Given that this is no longer occurring, it would be nice if the SDK at least offered an easy way to ship Base64 to the queue instead of raw strings.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage

[MFurn]

Is there a workaround for this? Azure Function Queue Triggers do not work with messages created by the the new SendMessage method in Azure.Storage.Queue because they are not encoded.
string messageAsJson = "Just Testing";
SendReceipt receipt= await queue.SendMessageAsync(messageAsJson);

The error message when the queue trigger picks up a queue message is:
Executed 'TestQueueFunction' (Failed, Id=2c1d8bf8-d288-4f1b-a64f-11ee9863c6d0) [3/30/2020 6:08:05 PM] System.Private.CoreLib: Exception while executing function: TestQueueFunction. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.CoreLib: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

[amishra-dev]

assigning it to you for tracking @jaschrep-msft

[MFurn]

Is there a way to set the encoding using the 12.3 queue client without the helper that the op is asking for? If not this seems more like a bug than a feature request since queue messages added by 12.3 can't be used in azure function queue triggers.

[speedy-ms]

What I am experiencing is that _sometimes_ the message is enqueued as Base64 and _sometimes_ it is not. It makes handling queue messages in Azure Functions triggers impossible as @MFurn has mentioned, and processing queue messages in a more manual way (via QueueClient.ReceiveMessagesAsync) much more difficult as you have to work out whether you need to Base64 decode the response or not before you proceed.

I agree with the sentiment that this is a bug rather than a feature request.

[JonathanHopeDMRC]

We are also seeing the behavior that speedy-ms is describing. Sometimes our messages are base64 encoded and sometime they are not. If there is a pattern as to what is getting encoded and not getting encoded it not obvious to me. I agree that this feels like a bug.

[speedy-ms]

Hi @JonathanHopeDMRC - I raised the following issue https://github.com/Azure/azure-sdk-for-net/issues/11358 and the advice from Microsoft was: "A quick workaround for you would be to base64 encode the string".

I have proactively Base64 encoded the string (and obviously always look to decode the message on the other end) and I am yet to see an instance where it is "double encoded".

So if you are looking for a quick workaround I can confirm that, while it is early days, this appears to work for us.

[amishra-dev]

We are also seeing the behavior that speedy-ms is describing. Sometimes our messages are base64 encoded and sometime they are not. If there is a pattern as to what is getting encoded and not getting encoded it not obvious to me. I agree that this feels like a bug.

@JonathanHopeDMRC the client library never encodes, if you are getting encoded messages, you are encoding it in your code.

[speedy-ms]

@amishra-dev or perhaps sometimes using the older library (Microsoft.Azure.Storage.Queue) to queue messages and the newer one to dequeue? I think in hindsight that must have been why I was under the impression that we "sometimes did" and "sometimes didn't" see base64 encoded messages - the ones with encoding were coming from older apps that hadn't updated yet.

[Jandev]

Is there a workaround for this? Azure Function Queue Triggers do not work with messages created by the the new SendMessage method in Azure.Storage.Queue because they are not encoded.
string messageAsJson = "Just Testing";
SendReceipt receipt= await queue.SendMessageAsync(messageAsJson);

The error message when the queue trigger picks up a queue message is:
Executed 'TestQueueFunction' (Failed, Id=2c1d8bf8-d288-4f1b-a64f-11ee9863c6d0) [3/30/2020 6:08:05 PM] System.Private.CoreLib: Exception while executing function: TestQueueFunction. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.CoreLib: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

I just stumbled across this issue after researching the issue I had. I'm using the Azure.Storage.Queues v12.3.2 packages.
My Azure Function is running on the Microsoft.Azure.WebJobs.Extensions.Storage v4.0.1 package and indeed see this message.

The message is also thrown in Visual Studio using the Cloud Explorer and navigate to a queue which holds messages sent via the Azure.Storage.Queues pacakge.

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

Learn more about known storage blob issues and solutions at https://go.microsoft.com/fwlink/?LinkId=532762.

The Microsoft Azure Storage Explorer can see the messages on the queue, also the Azure Portal doesn't has an issue with the messages sent via the mentioned package.

I don't know if it qualifies as a bug, but it certainly breaks lots of tools & other packages.

For now I've changed my code to

    await queueClient.SendMessageAsync(Base64Encode(serializedCommand), cancellationToken);

private static string Base64Encode(string plainText)
{
    var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
    return System.Convert.ToBase64String(plainTextBytes);
}

Hope this gets resolved soonish, either by updating the IntelliSense + docs or encoding by default (again?).

[shawndewet]

As a workaround I've implemented the following two Extension methods:

public static Azure.Response<Azure.Storage.Queues.Models.SendReceipt> SendMessage(this Azure.Storage.Queues.QueueClient queueClient, object value)
        {
            var plainText = JsonConvert.SerializeObject(value);
            var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
            return queueClient.SendMessage(Convert.ToBase64String(plainTextBytes));
        }

and

public static System.Threading.Tasks.Task<Azure.Response<Azure.Storage.Queues.Models.SendReceipt>> SendMessageAsync(this Azure.Storage.Queues.QueueClient queueClient, object value)
        {
            var plainText = JsonConvert.SerializeObject(value);
            var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
            return queueClient.SendMessageAsync(Convert.ToBase64String(plainTextBytes));
        }

[RakeshDPA]

Is there a workaround for this? Azure Function Queue Triggers do not work with messages created by the the new SendMessage method in Azure.Storage.Queue because they are not encoded.
string messageAsJson = "Just Testing";
SendReceipt receipt= await queue.SendMessageAsync(messageAsJson);

The error message when the queue trigger picks up a queue message is:
Executed 'TestQueueFunction' (Failed, Id=2c1d8bf8-d288-4f1b-a64f-11ee9863c6d0) [3/30/2020 6:08:05 PM] System.Private.CoreLib: Exception while executing function: TestQueueFunction. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.CoreLib: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

I just stumbled across this issue after researching the issue I had. I'm using the Azure.Storage.Queues v12.3.2 packages.
My Azure Function is running on the Microsoft.Azure.WebJobs.Extensions.Storage v4.0.1 package and indeed see this message.

The message is also thrown in Visual Studio using the Cloud Explorer and navigate to a queue which holds messages sent via the Azure.Storage.Queues pacakge.

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
Learn more about known storage blob issues and solutions at https://go.microsoft.com/fwlink/?LinkId=532762.

The Microsoft Azure Storage Explorer can see the messages on the queue, also the Azure Portal doesn't has an issue with the messages sent via the mentioned package.

I don't know if it qualifies as a bug, but it certainly breaks lots of tools & other packages.

For now I've changed my code to

    await queueClient.SendMessageAsync(Base64Encode(serializedCommand), cancellationToken);

private static string Base64Encode(string plainText)
{
    var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
    return System.Convert.ToBase64String(plainTextBytes);
}

Hope this gets resolved soonish, either by updating the IntelliSense + docs or encoding by default (again?).

This works for me. Thanks!

[speedy-ms]

A workaround is to base64 encode your queue message prior to placing on the queue, and then the handler will read it correctly.

On 26 Jun 2020 11:04 pm, Rakesh Pawar [e-mail] wrote:

Is there a workaround for this? Azure Function Queue Triggers do not work with messages created by the the new SendMessage method in Azure.Storage.Queue because they are not encoded.
string messageAsJson = "Just Testing";
SendReceipt receipt= await queue.SendMessageAsync(messageAsJson);
[image]https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F35820668%2F77958803-97c92900-72a3-11ea-8707-f31728117490.png&data=02%7C01%7Cdspeed%40machship.com%7C856e29ddc7484e35d28b08d819d17f82%7Ce48d99023156435789a3733c669c4339%7C0%7C0%7C637287734876854875&sdata=955%2By%2BvxHTHs1zy4dDNqlVKUAH%2Bo%2F7MWoNTCYBnR52s%3D&reserved=0
The error message when the queue trigger picks up a queue message is:
Executed 'TestQueueFunction' (Failed, Id=2c1d8bf8-d288-4f1b-a64f-11ee9863c6d0) [3/30/2020 6:08:05 PM] System.Private.CoreLib: Exception while executing function: TestQueueFunction. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.CoreLib: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

I just stumbled across this issue after researching the issue I had. I'm using the Azure.Storage.Queues v12.3.2 packages.
My Azure Function is running on the Microsoft.Azure.WebJobs.Extensions.Storage v4.0.1 package and indeed see this message.

The message is also thrown in Visual Studio using the Cloud Explorer and navigate to a queue which holds messages sent via the Azure.Storage.Queues pacakge.

The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
Learn more about known storage blob issues and solutions at https://go.microsoft.com/fwlink/?LinkId=532762https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D532762&data=02%7C01%7Cdspeed%40machship.com%7C856e29ddc7484e35d28b08d819d17f82%7Ce48d99023156435789a3733c669c4339%7C0%7C0%7C637287734876854875&sdata=W0BuE4uRvLqlmmqjVUmuyeZPWTXThy6SB65YwqM2UtY%3D&reserved=0.

[vs-non-base64-messages-on-poison-queue]https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F462356%2F84075822-0f6ab100-a9d5-11ea-8481-22c3a7297f22.png&data=02%7C01%7Cdspeed%40machship.com%7C856e29ddc7484e35d28b08d819d17f82%7Ce48d99023156435789a3733c669c4339%7C0%7C0%7C637287734876854875&sdata=KpwkYanlHqrR9CRlzdRVVUL3d8ubSHfBI9QxxNnEaHw%3D&reserved=0

The Microsoft Azure Storage Explorer can see the messages on the queue, also the Azure Portal doesn't has an issue with the messages sent via the mentioned package.

I don't know if it qualifies as a bug, but it certainly breaks lots of tools & other packages.

For now I've changed my code to

await queueClient.SendMessageAsync(Base64Encode(serializedCommand), cancellationToken);

private static string Base64Encode(string plainText)
{
var plainTextBytes = System.Text.Encoding.UTF8.GetBytes(plainText);
return System.Convert.ToBase64String(plainTextBytes);
}

Hope this gets resolved soonish, either by updating the IntelliSense + docs or encoding by default (again?).

This works for me. Thanks!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-sdk-for-net%2Fissues%2F10242%23issuecomment-650167623&data=02%7C01%7Cdspeed%40machship.com%7C856e29ddc7484e35d28b08d819d17f82%7Ce48d99023156435789a3733c669c4339%7C0%7C0%7C637287734876864867&sdata=x9y7uXWbKqPdch%2B92REBkeSEF2oOItJIsfaDnXeBuUE%3D&reserved=0, or unsubscribehttps://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALLRYFT3JGAQDOSEMZWSQC3RYSMGVANCNFSM4K72HLAA&data=02%7C01%7Cdspeed%40machship.com%7C856e29ddc7484e35d28b08d819d17f82%7Ce48d99023156435789a3733c669c4339%7C0%7C0%7C637287734876864867&sdata=l2Wj%2Fk8hzAseRAeUdW%2Ff140lyfYmdIsKnR0opfQgO2E%3D&reserved=0.

[jkotalik]

I hit this issue as well and was lost for quite a while. I think the documentation should be updated to mention that the message should be a base64 encoded UTF-8 string: https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/storage/Azure.Storage.Queues/src/QueueClient.cs#L1324.

[lukasvosyka]

I ran into the same issue. The fix was easy, but the fact that MS libs don't work out-of-the-box as they did with previous versions in conjunction with Azure is more than frustrating. Offloading problems to the customers of MS seems to be the solution to MS guys out there... 👎

[EntityAdam]

Frustrated that there appears to be problems all across the Azure SDKs. Followed the Get Started with Queues and it didn't work. Wound up here. I've had the same experience with Azure Storage Tables and those related packages as well.

[EntityAdam]

I'm going to go out on a limb here, I don't believe this is a BUG with the Azure SDKs but an limitation imposed with the input binding of the QueueTrigger in the Azure Functions (WebJobs SDK) requiring a base64 encoded response. source

It would call this a feature request for a switch for SendMessage() encoding. Worth noting, this is apparently already a feature in the Python SDK source issue.

[elliotchaim]

I also encountered this issue. Having previously used the older SDKs, this is very confusing.

[ChangeTheCode]

i also have this issue. I mirgrated the write / send part to the new package because nuget recommend it but it is not compatible to the nuget package Microsoft.Azure.Storage.Queue.
Of the reason which all before me described already. That's annoying in case you have distributed systems and you using storage account queues as interface
👎

[jeffw-wherethebitsroam]

I also encountered this issue after moving to Azure.Storage.Queues. I had no idea that my messages were previously being encoded as base64. Why would I? This was all hidden in the previous version of the client library and all I did was pass a UTF8 encoded message.

Old code for message insertion:

await queue.AddMessageAsync(new CloudQueueMessage(utf8msg));

New code:

await queue.SendMessageAsync(utf8msg);

How is the end user supposed to know that the encoding changed? Or more to the point, why does the end user even have to be aware that the encoding has changed? I put a UTF8 encoded message into the queue, I just want a UTF8 message out of the queue.

It looks like a bug in the WebJobs SDK that it does not support UTF8 encoded messages. This should not be a problem for the users of these libraries.

[podvlada]

Why is this still an issue? I also switched from old deprecated libraries to new Azure.Storage.Queues and could not why the Queues don't work until I found this thread.

[amishra-dev]

@podvlada I agree we should document this more prominently. Let me get started on that. @jaschrep-msft fyi.

[svengeance]

@amishra-dev
When I documented this issue I documented a separate documentation issue here - https://github.com/MicrosoftDocs/azure-docs/issues/49296

If this helps at all.

[iTiamo]

This is still an issue! Either implement having the SendMessage and SendMessageAsync methods convert to Base64 automatically, or mention in the documentation that you need to do this manually with Convert.ToBase64String...

[kasobol-msft]

This has been addressed in https://github.com/Azure/azure-sdk-for-net/pull/16689 along with base64 option and available in https://www.nuget.org/packages/Azure.Storage.Queues/12.5.0

[svengeance]

Good stuff. Will take a look at the release, thank you very much!

[imsam67]

Up to this point, I'd been using the Microsoft.WindowsAzure.Storage package and with this package, I'd simply send a MyQueueRequest object to my queue and everything worked fine.

Because the Microsoft.WindowsAzure.Storage package has been deprecated, I swithched to Azure.Storage.Queue and my Azure Function started throwing the following error:

Microsoft.Azure.WebJobs.Host: Exception binding parameter 'message'. System.Private.CoreLib: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

I've found this article that suggests that the new library requires JSON objects to be encoded in Base64 (https://briancaos.wordpress.com/2020/10/16/sending-json-with-net-core-queueclient-sendmessageasync/).

Up to this point, I actually never even serialized my MyQueueRequest object to JSON. The model binder took care of that for me automatically.

Does this mean, going forward, before sending the message to my queue, I need to first serialize MyQueueRequest object and then Base64 encode it and then reverse the process in my Azure Functions?

[kasobol-msft]

@imsam67 Could you please describe data flow in your system? I.e. what is producing and consuming messages, what versions of Storage SDK/Azure Function bindings are used by each component?
From description I'd guess that producer is sending message via Azure.Storage.Queue and consumer is Azure Function that uses older Storage SDK under the hood. Thus expecting base64 encoded message.

In general. V11 and lower Storage SDK as well as Azure Function bindings add base64 encoding (on send) and decoding (on receive) by default. That is independent from whether message was string, serialized json or bytes.

The V12 Storage SDK has changed default to not apply extra encoding/decoding and sends the message verbatim.

In mixed system (i.e. system components using different Storage SDK versions) this poses challenge as each component would have different default leading to inability to process messages exchanged between these components. In order to make it work one has to align the "base 64 behavior" between components.
Currently V12 users have two options. Either add code to encode/decode before/after calling SDK or use MessageEncoding property when creating QueueClients to set behavior globally.

[imsam67]

@kasobol-msft Our API app is the one that generates the messages and our Azure Functions app is the one that receives them. The API app was built in ASP.NET Core 3.1 and we've already upgraded it to .NET 5 so it's now an ASP.NET Core 5 app. Azure Functions app targets netcoreapp3.1 and is a v3 Azure Functions app.

As mentioned in my previous message, we were using Microsoft.WindowsAzure.Storage and never encoded/decoded our messages to Base64. After upgrading to Azure.Storage.Queue, we noticed that all our messages were ending up in the poison queue and that's when we realized the issue was due to Base64 encoding.

Interestingly, we fixed the issue by manually encoding messages in the API app before sending them out to our queue but we do NOT decode them in the Azure Functions app. Looks like the Azure Functions app is automatically decoding them. We initially tried decoding messages manually but that threw errors. When we stopped decoding them in the Azur Functions app, everything started working fine.

Because both the API and the Azure Functions app share common libraries that provide clients and repositories, we use Azure.Storage.Queue package in both API and Functions apps.