Azure-sdk-for-net: Determine EventData message size

[samneirinck]

Actual Behavior

Unable to determine the message size of a serialized EventData instance. I noticed this missing property while looking at the following StackOverflow question. There was a change related to batch sizes on Azure/azure-event-hubs-dotnet#79, but this issue is about a single EventData instance.

Expected Behavior

The old SDK has the property SerializedSizeInBytes (MSDN). Would like to have the same property available here.

Versions

• OS platform and version: Windows
• .NET Version: 4.7
• NuGet package version or commit ID: 1.1.0

If you're open for contributions I'd love to take a look.

[serkantkaraca]

Are you working with large messages? Why did you need to calculate the size of a single message?

Here is how the client calculates individual message size during batching.

    long GetSize(EventData eventData)
    {
        // Create AMQP message here. We will use the same message while sending to save compute time.
        var amqpMessage = AmqpMessageConverter.EventDataToAmqpMessage(eventData);
        AmqpMessageConverter.UpdateAmqpMessagePartitionKey(amqpMessage, this.PartitionKey);
        eventData.AmqpMessage = amqpMessage;
        return eventData.AmqpMessage.SerializedMessageSize;
    }

[samneirinck]

Yes, I'd like to know when we have to store the message on blob and pass a reference in the message on event hub.

The AmqpMessageConverter class, among others, is internal, so I'm not sure it's really easy to get the size from externalm not able to access it as the nuget consumer.

[serkantkaraca]

Will you be saving event data object or the payload only? If you save the payload you can manage your own size calculation and there won't be any overhead due to message serialization and AMQP headers.

In order to decide whether to send or save you can just define a safe body size say 260KB which will give you enough overhead for some message properties. Please let me know how this sounds.

[samneirinck]

I'd like to save both the payload and its properties, because the properties contain correlation information useful for logging.

I'm currently using the solution you mentioned, adding some space for overhead of message properties. While searching for a better solution I stumbled upon the `SerializedSizeInBytes property of the previous nuget package, which would allow me to be 100% correct and not assume the serialized size of my properties will be below whatever threshold I currently have. Was there a specific reason to no longer provide access to the serialized message size?

[serkantkaraca]

Probably that one lost in the translation. I will mark this as a feature request. We can provide it in the near future.

[erick-thompson]

This would be useful for me as well, as I need to provide information back to the caller about the event(s) that caused that issue.

[erick-thompson]

In addition to getting the sizes of messages, which is great for validation prior to sending, the error case(s) would be nice to cover as well.

Currently, the same exception is throw for an individual message being too large and for a group of messages being too large. These are two different error conditions, with different options.

When an individual message is too large, there is nothing that can be done. It's a fatal error, and should be reported back to the caller as a fatal error.

When a group of messages is being sent, and the total batch size is too large, you can make the event batches smaller and submit multiple batches. It's not a fatal error (as long as no individual message is too large).

Right now, a MessageSizeExceededException doesn't indicate which error case is being hit. It would be better to have a second exception (MessageBatchSizeExceededException). Then I can take the appropriate action in the catch block.

[serkantkaraca]

Sorry I didn't have time to work on this feature request so far. This one will be part of next SDK release.

[davidrevoledo]

@serkantkaraca do you want I try to take a look to this ?

[serkantkaraca]

@serkantkaraca do you want I try to take a look to this ?

Sure

[serkantkaraca]

Keep in mind that you need to consider partition key while calculating message size.

[mendesbarreto]

Hello everybody.

We have a evolution from this enhancement?
I am new on EventHubs and I really trying to understanding the billing system. I got 365314 messages witch my client sent to Eventhub but 4 millions on console from Azure.
(When I tryed to download the messages it has only 365314).

So I am trying to eliminate what is causing this high amount of data, in that case I am suspecting of the size of message is passing the 64kb and this method will help my analysis for sure.

[serkantkaraca]

Hello everybody.

We have a evolution from this enhancement?
I am new on EventHubs and I really trying to understanding the billing system. I got 365314 messages witch my client sent to Eventhub but 4 millions on console from Azure.
(When I tryed to download the messages it has only 365314).

So I am trying to eliminate what is causing this high amount of data, in that case I am suspecting of the size of message is passing the 64kb and this method will help my analysis for sure.

Your issue doesn't sound like SDK related. Please open a support request.

[mendesbarreto]

Hello everybody.
We have a evolution from this enhancement?
I am new on EventHubs and I really trying to understanding the billing system. I got 365314 messages witch my client sent to Eventhub but 4 millions on console from Azure.
(When I tryed to download the messages it has only 365314).
So I am trying to eliminate what is causing this high amount of data, in that case I am suspecting of the size of message is passing the 64kb and this method will help my analysis for sure.

Your issue doesn't sound like SDK related. Please open a support request.

Not necessarily, I just gave an introduction of what I would like to do with this property.

This property will help me a lot to investigate something or to insert into my logs and identify possible giant messages to launch warnings.

[serkantkaraca]

Please reconsider creating a feature request for Track 2 SDK - https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/eventhub/Azure.Messaging.EventHubs