Azure-sdk-for-net: [Event Hubs] How can I check the message size when using Azure IoT Hub?

[Marusyk]

Query/Question
I'm using standard IoT Hub SDKs to read messages from built-in endpoint of IoT Hub.

How can I get a full message size for analytics? I mean not only Data size (see code) but a full message size with properties.
Size that actually counted by IoT Hub.

    private async Task ProcessMessage(ProcessEventArgs eventArgs)
    {
        string equipmentNumber = eventArgs.Data.SystemProperties["iothub-connection-device-id"].ToString();
        string message = Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray());
        int bodySize = message.Length;

[jsquire]

Unless I'm mistaken, the structure of your code seems to indicate use of the Event Hubs library to process messages; that signature corresponds to the EventProcessorClient handler, I believe. If I'm not correct, would you please be so kind as to share which library and which version the snippet corresponds to?

Assuming that I'm correct, there is no API within the client library to measure the size of an event in bytes. There is no public contract that specifies how either the Event Hubs or IoT Hub services store messages internally. The closest representation would be what the size of the message is when serialized to AMQP format for transport. It would be possible to measure the size of incoming messages before deserializing and propagate that, potentially.

If you're willing, I'd be interested in understanding more about your scenario and how this information would be useful to your application.

[Marusyk]

Hello @jsquire

Thank you for your response.

Unless I'm mistaken, the structure of your code seems to indicate use of the Event Hubs library to process messages; that signature corresponds to the EventProcessorClient handler, I believe.

Yes, you are right!

I'm using standard IoT Hub SDKs to read messages from built-in endpoint of IoT Hub.

Azure.Messaging.EventHubs 5.1.0
Azure.Messaging.EventHubs.Processor 5.1.0

It would be possible to measure the size of incoming messages before deserializing and propagate that, potentially.

Do you mean that this is correct, right?

string message = Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray());
int bodySize = message.Length;

If you're willing, I'd be interested in understanding more about your scenario and how this information would be useful to your application.

I'm going to send the real size of each message to DataLake storage. Then, the data will be used to calculate how many Megabytes a device send per day, hour, etc. In addition, it will help us detect if the messages exceed (or almost exceed) the size of the IoT Hub messages limits

[jsquire]

Do you mean that this is correct, right?

No, I meant that the client library has access to the actual size of the message, as it was sent for transport. That includes all of the AMQP structure, overhead, and such. That also includes some properties that are owned by the broker and populated by the service when messages are requested by consumers. However, I don't believe that set of data would satisfy your requirement of understanding the IoT Hub message limits.

Just to be sure that I understand, I think what you're looking to do is understand the total size of all outgoing messages for a given device per unit of time. Did I interpret that correctly? Assuming so, can you tell me how you're sending messages from the IoT devices? My suspicion is that there's a better way to measure the metric that you're after using the IoT Hub service, but I don't know it well - so I'd like to make sure that I have good context from the sender side and then see if I can get some insight from the IoT folks.

[jsquire]

//cc: @drwill-ms - Any chance that you may have some thoughts on the best way to gather metrics on messages send to IoT Hub, in the context of a single device?

[drwill-ms]

I'm not sure what all the auxiliary requirements are, but one might set up a hub message routing rule to send them to a storage account, and then look at the size, on disk, of the message there.

[jsquire]

I'm not sure what all the auxiliary requirements are, but one might set up a hub message routing rule to send them to a storage account, and then look at the size, on disk, of the message there.

Thanks for your help, @drwill-ms . Follow-up questions, if you would be so kind: wouldn't the proposed approach also include the broker-owned system properties? Are those counted towards the IoT Hub quotas? Is there a means to capture the size of a message during send - and would that be an accurate measurement for calculating quotas?

[drwill-ms]

I don't know, for sure. We'd have to try it, but it may be a better question for the IoT Hub service team.

The quotas, from what I know, are related to number of messages sent, not the size.

There is a size limit in sending, which includes the body, custom properties, and system properties. I've noticed the SDKs don't have a way to calculate that info, and it has been on my list of things to look at when we have the time. In practice, I found I could send many, many messages without even coming close to the limit, so it seemed like a low priority.

[Marusyk]

Thank you all for your responses.

I just need to get the size of enqueued message, not only Data content but whole with all properties.

I don't believe that set of data would satisfy your requirement of understanding the IoT Hub message limits.

I don't need to understand the IoT Hub message limits, I just need some statistics about data usage

If there is no way, I will collect the size of Data part of a message.

[drwill-ms]

I've logged the feature request in our github repo for you. We'll see if we can get to that before the end of the year.

[Marusyk]

Thank you very much, I really appreciate it. Thanks.

[jsquire]

Thanks very much for your help, @drwill-ms. @Marusyk, is there anything further on this that you'd like to discuss, or are you amenable to me closing this out in favor of the issue that David linked?