Azure-webjobs-sdk: Deserializing ServiceBusMessage to POCO fails

[ChristianWeyer]

My setup:

• macOS 10.13.6
• VS Code Version 1.28.2
• Azure Functions Core Tools (2.1.725 Commit hash: 68f448fe6a60e1cade88c2004bf6491af7e5f1df)
• Function Runtime Version: 2.0.12134.0

Repro is here: https://github.com/ChristianWeyer/azure-functions-servicebus-serialization-repro

This is the scenario:

C# Azure Function 1:
-HTTP Trigger -> creates a message and sends it into SB queue via output binding
=>
Java Azure function:
-SB queue trigger -> creates new message based on incoming message and sends it to another queue via SB queue output binding
=>
C# Azure Function 2:
-Service Bus queue trigger -> inspects received message... and does nothing ;-)

The message coming from the Java function into C# Function 2 (via SB) cannot be deserialized through model binding - so this fails:

public static class NotifyClientsAboutOrderShipmentFunctions
{
    [FunctionName("NotifyClientsAboutOrderShipment")]
    public static void Run(
        [ServiceBusTrigger("shippingsinitiated", Connection = "ServiceBus")]
        ShippingCreatedMessage msg, // Issue with deserializing incoming message
        ILogger log)
    {
        log.LogInformation($"NotifyClientsAboutOrderShipment SB queue trigger function processed message: {msg}");

        var messageToNotify = new { orderId = msg.OrderId };

        // TODO: Notify users' clients through SignalR...
    }
}

Exception:

[02/11/2018 09:06:21] Executing 'NotifyClientsAboutOrderShipment' (Reason='New ServiceBus message detected on 'shippingsinitiated'.', Id=9b058fc4-b60b-4e50-aa1b-c10f97135767)
[02/11/2018 09:06:24] Executed 'NotifyClientsAboutOrderShipment' (Failed, Id=9b058fc4-b60b-4e50-aa1b-c10f97135767)
[02/11/2018 09:06:24] System.Private.CoreLib: Exception while executing function: NotifyClientsAboutOrderShipment. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'msg'. System.Private.DataContractSerialization: There was an error deserializing the object of type Serverless.Messages.ShippingCreatedMessage. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted.
[02/11/2018 09:06:24] MessageReceiver error (Action=UserCallback, ClientId=MessageReceiver1shippingsinitiated, EntityPath=shippingsinitiated, Endpoint=cw-serverless-microservices.servicebus.windows.net)
[02/11/2018 09:06:24] System.Private.CoreLib: Exception while executing function: NotifyClientsAboutOrderShipment. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'msg'. System.Private.DataContractSerialization: There was an error deserializing the object of type Serverless.Messages.ShippingCreatedMessage. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted.

Not too helpful with the missing details 😉

If I change the function to look like this (and do the deserialization manually), it works:

public static class NotifyClientsAboutOrderShipmentFunction
{
    [FunctionName("NotifyClientsAboutOrderShipment")]
    public static void Run(
        [ServiceBusTrigger("shippingsinitiated", Connection = "ServiceBus")]
        /*ShippingCreatedMessage*/ string msg, // Issue with deserializing incoming message
        ILogger log)
    {
        log.LogInformation($"NotifyClientsAboutOrderShipment SB queue trigger function processed message: {msg}");

        ShippingCreatedMessage message = JsonConvert.DeserializeObject<ShippingCreatedMessage>(msg);

        var messageToNotify = new { orderId = message.OrderId };

        // TODO: Notify users' clients through SignalR...
    }
}

This is the message being sent via the 'response' queue:

{
  "Id": "c497b524-5929-48e5-aa7e-1a5992678afb",
  "Created": "2018-11-02T09:15:20.000133Z",
  "OrderId": "9190338c-546d-4d44-a7f3-081d6563bfc8"
}

What is expected:

1. Automatic deserialization should properly work
2. If it does not work, a more detailed & more helpful exception message should be shown why it did not work

Thanks!

[fabiocav]

@ChristianWeyer thank you for the details!

Just want to make sure, that is the exact payload going into the shippingsinitiated queue (not response), correct? I'm assuming you're also able to repro this if you manually queue a message with that payload, is that correct?

[ChristianWeyer]

Yep, just checked by manually sending above snippet via the https://portal.serverless360.com/ tooling.

[ChristianWeyer]

Any plans how to proceed with this @fabiocav ? :-)

[fabiocav]

This has unfortunately not been assigned to a sprint yet, but I did have a chat with @pragnagopa about this (again, the detailed repro was helpful!). I think that splitting this up to tackle option 2 quickly would help, that way we can at least improve the experience until a more permanent fix is in place.

@pragnagopa thoughts?

[ChristianWeyer]

Let me ping here again... :-)
Thanks @pragnagopa !

[pragnagopa]

Apologies for the delayed response. The issue reproes standalone when using a servicebustrigger.

using System;
using System.Threading.Tasks;

public static void Run(ShippingCreatedMessage myQueueItem, ILogger log)
{
    log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
}

public class ShippingCreatedMessage
{
    public Guid Id { get; set; }
    public DateTime Created { get; set; }
    public Guid OrderId { get; set; }
}

Trigger this function with payload:

// Create a new message to send to the queue

 string messageBody = "{ \"Id\": \"c497b524-5929-48e5-aa7e-1a5992678afb\",   \"Created\": \"2018-11-02T09:15:20.000133Z\",   \"OrderId\": \"9190338c-546d-4d44-a7f3-081d6563bfc8\"}";

 var message = new Message(Encoding.UTF8.GetBytes(messageBody));

   // Write the body of the message to the console
 Console.WriteLine($"Sending message: {messageBody}");

 // Send the message to the queue
  await queueClient.SendAsync(message);

Exception is coming from here:

https://github.com/Azure/azure-webjobs-sdk/blob/6930c4ea9ec78db12a05a2d0bf38cce42ab04e13/src/Microsoft.Azure.WebJobs.Extensions.ServiceBus/Triggers/MessageToStringConverter.cs#L46-L57

To deserialize to a POCO, Webjobs sdk needs to convert to the requested type and not string. More details here: https://abhishekrlal.com/2012/03/30/formatting-the-content-for-service-bus-messages/

I will move this issue to webjobs sdk repo. Thanks again @ChristianWeyer for detailed repro.

[pragnagopa]

cc @mathewc
Updated the title to reflect the issue.

[SeanFeldman]

Does this have a milestone when to expect the fix?

[alrod]

@SeanFeldman, we are expecting POCO object desirialized as xml by default. If you want to send JSON please add "application/json" ContentType to your message.

https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Extensions.ServiceBus/Triggers/UserTypeArgumentBindingProvider.cs#L74

[alrod]

https://github.com/Azure/azure-webjobs-sdk/pull/2209

[SeanFeldman]

This looks rather odd, but I'll check it, @alrod

[SeanFeldman]

@alrod JSON-serialized messages sent using the new client (Microsoft.Azure.ServiceBus) do work.
When it comes to XML, there I think you have a mistake. The .GetBody<TInput>() is an extension method to take BrokeredMessages serialized using XmlObjectSerializer and deserialize those. It has nothing to do with XML serialization. If a message is sent by the new ASB client and it was a valid XML content, .GetBody<T>() will fail to deserialize it with the following exception:

System.Private.DataContractSerialization: There was an error deserializing the object of type PlaceOrder. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted.

Which means that this is not about XML serialization, but about XmlObjectSerialization. And that it will only handle messages sent with the old ASB client (WindowsAzure.ServiceBus) in the format of TInput to be able to be deserialized properly.

[SeanFeldman]

There's a logical error with UserTypeArgumentBindingProvider. The non-JSON part (under else) is not designed to work with the new client and deserialize XML based content.

[SeanFeldman]

@ChristianWeyer deserialization to POCO does work if

1. You specify ContentType header with application/json as @alrod pointed out
2. Send the message using the new ASB client

[ChristianWeyer]

Thanks @SeanFeldman - this is not what users expect if they have code on the sender side that just uses a POCO, as well.

In my original case, I had a Java Azure Function sender with a POJO, and a .NET Azure Function receiver with the same POCO.
This fails. And this should 'just work', IMHO //.cc @alrod

[alrod]

@alrod JSON-serialized messages sent using the new client (Microsoft.Azure.ServiceBus) do work.
When it comes to XML, there I think you have a mistake. The .GetBody<TInput>() is an extension method to take BrokeredMessages serialized using XmlObjectSerializer and deserialize those. It has nothing to do with XML serialization. If a message is sent by the new ASB client and it was a valid XML content, .GetBody<T>() will fail to deserialize it with the following exception:

System.Private.DataContractSerialization: There was an error deserializing the object of type PlaceOrder. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted.

Which means that this is not about XML serialization, but about XmlObjectSerialization. And that it will only handle messages sent with the old ASB client (WindowsAzure.ServiceBus) in the format of TInput to be able to be deserialized properly.

@SeanFeldman you are right. I closed the PR.

[SeanFeldman]

This does raise a question about the design of this class.
I.e. how to deserialize an XML serialized POCO sent using the new client.

[alrod]

@SeanFeldman, ServiceBusTrigger supports only JSON and XmlObjectSerializer deseralizations. you can deserialize XML serialized POCO manually in the function code.

[alrod]

PR: https://github.com/Azure/azure-webjobs-sdk/pull/2211

[SeanFeldman]

@alrod in the future you could add fixes #2154 in the PR description and it would

1. Close this issue when PR is merged
2. Link to the issue, as well as add a link back from the issue to the PR so that you don't have to do it manually (GitHub feature)

[alrod]

https://github.com/Azure/azure-webjobs-sdk/commit/97e9d26856d039e259b04e1229069141f7fb0348