Azure-sdk-for-net: [BUG] Service Bus - OperationCanceledException in exception received handler while closing the message receiver

[satano]

Describe the bug
I have a QueueClient for receiving messages (basically using it just as MessageReceiver). It is working as expected - the messages are being processed. When I want to stop receiving messages, I call QueueClient.CloseAsync(). There are still messages in the queue, so I will stop the receiver _in the middle of the work_. So the receiver/client is in the closing state, and I (almost) alway get an error in my ExceptionReceivedHandler - it is OperationCanceledException with the simple message "_The operation was canceled._".

Exception or Stack Trace
I do not have a direct exception. Just the exception from ExceptionReceivedEventArgs. The call stack in the received exception is:

at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at Microsoft.Azure.ServiceBus.MessageReceivePump.MessagePumpTaskAsync()

To Reproduce
Just stop MessageReceiver while there still are messages in the queue to process and look in your exception handler.

Expected behavior
I am not really sure about it. But I think that if the client is in closing state, I should not get this type of error in my error handler (ExceptionReceivedHandler). It is just annoying, because basically, I will get this error every time I am stopping my receiver.

Setup (please complete the following information):

• OS: Windows 10
• IDE : Visual Studio 2019
• Version of the Library used: Microsoft.Azure.ServiceBus 3.4.0

Additional context
I attached the source of Microsoft.Azure.ServiceBus to dig deeper what is going on.

• The system throws while trying to get semaphore lock. The receiver is in closing state: pumpCancellationToken.IsCancellationRequested == true. So the semaphore throws OperationCanceledException.
• In the catch block, there is info about not propagating ObjectDisposedException when the pump is stopping.
• I think, that this situation with OperationCanceledException is the same and it should not be reported too.
• The final note is, that it later comes into the finally block. The condition message == null is true, because loading of the message was skipped. And so it enters the if and releases semaphore lock. I am not sure about this, but I think the lock was not taken (semaphore threw exception), so probably it should not be released.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [x] Bug Description Added
• [x] Repro Steps Added
• [x] Setup information Added

[kurtzeborn]

Thank you for opening this issue! We are routing it to the appropriate team for follow up.

[hiramhuang]

I meet a similar issue when I use the extension for Azure Functions.

Microsoft.Azure.WebJobs.Extensions.[email protected]

System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Azure.WebJobs.ServiceBus.SessionMessageProcessor.CompleteProcessingMessageAsync(IMessageSession session, Message message, FunctionResult result, CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener.ProcessSessionMessageAsync(IMessageSession session, Message message, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.SessionReceivePump.MessagePumpTaskAsync(IMessageSession session)

[jsquire]

@nemakam, @binzy: Would one of you be so kind as to offer your thoughts?

[satano]

I can (try to) help with the PR if you guys confirm this.

[Jtmw11]

Is there any existing work around for this?

[Jtmw11]

For context I running .NET Core 2.1(I tried 2.2 as well just for kicks) in an ASP.NET Core Web App and I continue to run into this error

Message handler encountered an exception System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int32 millisecondsTimeout, CancellationToken cancellationToken) at Microsoft.Azure.ServiceBus.MessageReceivePump.MessagePumpTaskAsync(). Exception context for troubleshooting:
`- Endpoint: xxxxxxxxxx.servicebus.windows.net

• Entity Path: xxxxxxx/Subscriptions/SubscriptionB
• Executing Action: Receive`

I have run this exact same code in an .NET Core Console Application and the code works perfectly. If there is any more information I can give please let me know!

[Jtmw11]

Solved..

In the MSDN example, there is a closeAsync() that is immediately following setting up the message handler. This isn't the appropriate place when using the .NET implementation. That was the cause of my premature Cancellation. The MSDN exmaple seems to be incorrect.

[axisc]

@Jtmw11, could you provide a link to the example you are referring to? I can go ahead and rectify it.

[jfggdl]

@Jtmw11, would you please share the link to the example you are referring to? We would like to fix any documentation/example that may have problems.

[Jtmw11]

My apologies guys, after looking at it again. I used the following instructional.

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-how-to-use-topics-subscriptions

However I implemented it into a web app format. The issue I faced was that the console.Readline wasn't doing exactly as I had expected. Essentially each time I was opening the message handler I was closing it with the respective closeasync(). That is a misunderstanding on my end, I had forgotten it was a console based example.

[jfggdl]

@Jtmw11, great that you identified the problem. I am going to close this issue up, but add any questions or comments and we can reopen this issue.

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[alrod]

Hi @jfggdl, I have repro. It looks like I can repro only with latest 4.0.0 ServiceBus SDK.

Can you please repopen that bug?

using Microsoft.Azure.ServiceBus;
using Microsoft.Azure.ServiceBus.Core;
using System;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace SBTest
{
    class Program
    {
        static async Task Main(string[] args)
        {
            Console.WriteLine("Started");

            string conectionString = "xxx";
            string entityName = "xxx";

            MessageReceiver receiver = new MessageReceiver(conectionString, entityName);
            QueueClient client = new QueueClient(conectionString, entityName);
            MessageHandlerOptions options = new MessageHandlerOptions(ExceptionReceivedHandler);
            receiver.RegisterMessageHandler(ProcessMessageAsync, ExceptionReceivedHandler);

            await client.SendAsync(new Message(Encoding.UTF8.GetBytes("test")));
            await Task.Delay(10000);

            await client.CloseAsync();
            await receiver.CloseAsync();

            await Task.Delay(10000);

            Console.WriteLine("Finished");
            Console.ReadLine();
        }

        static async Task ProcessMessageAsync(Message message, CancellationToken cancellationToken)
        {
            Console.WriteLine("Message processed");
        }

        static Task ExceptionReceivedHandler(ExceptionReceivedEventArgs eventArgs)
        {
            Console.WriteLine("Exception: " + eventArgs.Exception);
            return Task.CompletedTask;
        }
    }
}

Output:

Message processed
Exception: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeoutAsync(TaskNode asyncWaiter, Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.Azure.ServiceBus.MessageReceivePump.MessagePumpTaskAsync()
Finished

same issue: https://github.com/Azure/azure-service-bus-dotnet/issues/622

[manishthanneeru]

Have the same issue, guess will have to work with v3.4.0 until this is fixed

[msftbot[bot]]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[inkel]

I've ported an application from .NET Core 2.2 to 3.0 and I'm getting this exact same error. Any advice on how to proceed?

[satano]

@inkel This is not about version of .NET Core, but about version of ServiceBus Library. According to the merged PR above your comment, the fix will be in version 4.1.1 of Microsoft.Azure.ServiceBus. Current published version is 4.1.0. So you have to wait for the next release.

[joeyeng]

@inkel This is not about version of .NET Core, but about version of ServiceBus Library. According to the merged PR above your comment, the fix will be in version 4.1.1 of Microsoft.Azure.ServiceBus. Current published version is 4.1.0. So you have to wait for the next release.

Is there a way to get notified of Microsoft.Azure.ServiceBus releases specifically instead of getting notifications for everything in the azure-sdk-for-net repo?

[inkel]

Thank you, @satano.

[nemakam]

@jsquire do you know a way to get this - https://github.com/Azure/azure-sdk-for-net/issues/6410#issuecomment-551210267

[SeanFeldman]

This is something that would need to be implemented on the GitHub side. Alternatively, you can create a filter in your inbox or subscribe to a NuGet notification with a feed I've created https://libraries.io/nuget/Microsoft.Azure.ServiceBus/versions.atom. I'm using IFTTT with the feed to notify in Slack @nemakam.

[nemakam]

@joeyeng, FYI. The next version of 4.1.1 is released now. :)

[joeyeng]

@joeyeng, FYI. The next version of 4.1.1 is released now. :)

Ha, yea I saw, thanks! 👍🏼