Azure-sdk-for-net: [QUERY] Azure.Messaging.ServiceBus Processing speed of messages question

[chris-skuvault]

Query/Question
How can we help?

I have been testing using Azure.Messaging.ServiceBus recently and have noticed slow processing speeds when using session message processor or any of the Session Receiver ReceiveMessagesAsync overloads and just looping through all my messages. So far in my testing I have only been able to make a single receiver process between 5 and 8 messages a second when receiving 5000 messages that are 100 byte per message. When running a lot of session receivers concurrently processing different messages session with a small number of messages the performance is pretty good but a single receiver with a lot of messages doesn't seem great. I've also played around with the prefetch option and maybe saw a message or two a second improvement. I've tried prefetch values of 100 and 500.

Is this excepted or am I doing something wrong here?

Here is a sample of my code to receive just as a basic example. This uses the IAsyncEnumerable implmentation but I get similar speeds when using the other overload of ReceiveMessagesAsync and when using the session processor itself.

````
var receiver = await _serviceBusClient.CreateSessionReceiverAsync(,
, new ServiceBusSessionReceiverOptions
{
SessionId = "1",
}, token);

var messageCount = _consoleOptions.NumberOfMessagesPerSession;

await foreach (var message in receiver.ReceiveMessagesAsync(token))
{
messageCount--;
await receiver.CompleteMessageAsync(message, token);

if (messageCount == 0)
{
    break;
}

}
````

Environment:

• Azure.Messaging.ServiceBus-preview 4
• .net core 3.1

[AlexGhiondea]

@JoshLove-msft can you take a look?

[JoshLove-msft]

@chris-skuvault when operating on a single session you are constrained to a single session link as that link will lock the session. Adding concurrent processing of an individual session (available in upcoming preview), would help if your bottleneck is the time it takes to perform processing within your application. It won't, however, increase the throughput of messages that can travel over the link as we are still constrained by the single link. Using Prefetch will help maximize this throughput with the caveats that if your application doesn't process a message that has been prefetched, the delivery count will still get incremented even though your receiver didn't receive it.

I will run some tests locally to see if my numbers are way different than yours.

[chris-skuvault]

@JoshLove-msft The concurrent processing of an individual session in the next preview will definitely help performance in that regard. I don't think I will have issues in that sense as I will only ever be able to have a session call concurrency of 1 so things process in order.

I'm trying to get an understanding on how fast a single session can be processed from a queue with a call concurrency of 1. So if my session queue has 5k messages on it I will get an idea of how long that might take to process out of the queue. 5-8 messages a second seems like it is a little slower than I expected so I'm wondering if I might be doing something incorrectly.

I have tried on a regular non session queue with the same receiver code above for 5k messages and get similar speeds as with a session so maybe it is what it is.

Thanks for your help as always!

[JoshLove-msft]

So I have tested this out and I am seeing roughly the same numbers that you reported (around 5-10 msgs/sec) using the same code. We have a ReceiveMessagesAsync overload that takes a maxMessageCount parameter. This allows you to receive a set of messages at a time. However, even when using this overload I didn't see much change. The next thing I tried was commenting out the message completion. This makes the throughput skyrocket up to over 100 msgs/sec when using a batch size of 10. When receiving one message at a time with no message completion I was seeing about 15 msgs/sec. So this tells me that message completion (and message settlement in general - I also tried abandoning the message) is the bottleneck.

If I receive 10 messages at a time, and complete them all in parallel, I am seeing throughput of around 25 msgs/sec. If you want to increase the throughput, and you are okay with not completing them sequentially, I think something like this is a good option:

```c#
while (messageCount > 0)
{
var msgs = await receiver.ReceiveMessagesAsync(10);
messageCount -= msgs.Count;
var tasks = new List();
foreach (var msg in msgs)
{
tasks.Add(receiver.CompleteMessageAsync(msg));
}
await Task.WhenAll(tasks);
}

If you need to ensure that each batch of 10 messages completes as an atomic unit, you can consider using [transactions](https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-transactions). Using transactions like this only works when using a session receiver directly and would not really be possible when using the processor model, as there wouldn't be a way to create a transaction spanning all of the concurrent callbacks for a session.

```c#
while (messageCount > 0)
    var msgs = await receiver.ReceiveMessagesAsync(10);
    messageCount -= msgs.Count;
    var tasks = new List<Task>();

    using (var ts = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
    {
        try
        {
            foreach (var msg in msgs)
            {
                tasks.Add(receiver.CompleteMessageAsync(msg));
            }
            // if this throws the transaction will not be completed
            await Task.WhenAll(tasks);
            ts.Complete();
        }
        catch
        {
            break;
        }
    }
}

[slav]

Hi @JoshLove-msft. Unfortunately completing messages in batch is not acceptable in our scenario. We process one message at a time and need to complete it as soon as it processed. Batching up would make it very difficult for us to maintain certain gurantees about message processing consistency in the face of application failures and crashes. It would require a lot of work and testing on our part. Doing that will make pretty much undo any benefit of using cloud service.

From our preliminary tests it's CompleteMessage that's very slow. Each CompleteMessage takes around 150-300 ms. That's extremely slow, way slower that Azure Queue for example.

Do you have any idea why it's so slow and any other ideas on what can be done to improve the speed? Could it be something inside the library causing it, not the Service Bus itself?

[JoshLove-msft]

@slav - agreed that message settlement is the bottleneck - I'm seeing times of between 100-150 ms per CompleteMessageAsync call. I tested this out using the Track 1 library and am seeing roughly the same results. Is there a particular benchmark that you are trying to hit?

[slav]

I would expect no more than 10ms to complete message, preferably under 5ms.

[JoshLove-msft]

@axisc are there any published benchmarks on message settlement response times?

[JoshLove-msft]

@chris-skuvault did you have any more questions here? Otherwise I will go ahead and close out this issue. Thanks!

[chris-skuvault]

@JoshLove-msft Think i'm good for now. Thanks for all your assistance!

[chris-skuvault]

@chris-skuvault when operating on a single session you are constrained to a single session link as that link will lock the session. Adding concurrent processing of an individual session (available in upcoming preview), would help if your bottleneck is the time it takes to perform processing within your application. It won't, however, increase the throughput of messages that can travel over the link as we are still constrained by the single link. Using Prefetch will help maximize this throughput with the caveats that if your application doesn't process a message that has been prefetched, the delivery count will still get incremented even though your receiver didn't receive it.

I will run some tests locally to see if my numbers are way different than yours.

Hey @JoshLove-msft ,

I was doing some testing locally with session receivers and the Prefetch option and I'm not able to produce the behavior you have mentioned here. My test is setup as follows I send 10 messages to service bus queue. I start my receiver with a PrefetchCount of 50. I'm receiving with 1 maxMessage per receive operation. After receiving 6 messages I stop the receiver then inspect the remaining 4 with the expectation that those remaining 4 messages have an incremented delivery count but that doesn't seem to be the case. Has this functionality changed since this was written or am I doing something incorrectly? Thanks!

[JoshLove-msft]

I think this might be a difference in the behavior when using sessions:
https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-sessions#impact-of-delivery-count

The delivery count is apparently only incremented if the session lock is lost.