Azure-sdk-for-net: No active Transaction was found for ID

[SeanFeldman]

Describe the bug

When performing transactional processing (using TransactionScope and send-via feature) of messages concurrently, using multiple message senders, the System.InvalidOperationException exception is thrown.

Exception or Stack Trace

System.InvalidOperationException: No active Transaction was found for ID 'txn:9e879b829a0e446d97d6ac7625862833:9_G5'. The Transaction may have timed out or attempted to span multiple top-level entities such as Queue or Topic. The server Transaction timeout is: 00:02:00. Reference:fbcf536a-d67a-4a0f-8fb7-7532b8360f78, TrackingId:44c4b18d00000014000024ca5c5832b5_G5_B7, SystemTracker:SOMSYSTEMNAME:Queue:SOMEQUEUENAME, Timestamp:2019-02-04T12:40:21
   at Microsoft.Azure.ServiceBus.Core.MessageSender.OnSendAsync(IList`1 messageList)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.RetryPolicy.RunOperation(Func`1 operation, TimeSpan operationTimeout)
   at Microsoft.Azure.ServiceBus.Core.MessageSender.SendAsync(IList`1 messageList)
   at NServiceBus.TransportReceiveToPhysicalMessageProcessingConnector.Invoke(ITransportReceiveContext context, Func`2 next)
   at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext)
   at NServiceBus.Transport.AzureServiceBus.MessagePump.ProcessMessage(Task`1 receiveTask)

To Reproduce

Repro code is located in https://github.com/SeanFeldman/TransactionIssueRepro, TransactionIssueRepro.sln

• Sender project is seeding the receiver's queue with messages
• Receiver project is logging message processing.
• For each processed message, ten outgoing messages are emitted

Code Snippet

See the previous section

Expected behavior

There should be no exceptions.

Screenshots

N/A

Setup (please complete the following information):

• OS: Windows 10 Pro 1903
• IDE : VS 2019 16.1.3
• Azure Service Bus 3.4.0 / netcoreapp2.2

Additional context

When concurrency is set to one and a single outgoing MessageSender is used, this exception is not thrown. Whenever concurrency is higher than one, the exception is thrown.

Originally reported by NServiceBus customers using Azure Service Bus transport here. My investigation led to the native repro w/o NServiceBus code in it.

All NServiceBus customers on ASB transport with transactions enabled are affected 🐼

[SeanFeldman]

cc @nemakam

[SeanFeldman]

With the monolithic repository, there's no way to indicate what client is affected w/o polluting the title of the issue, which I didn't want to happen. This is Azure Service Bus client related issue.

[SeanFeldman]

@AlexGhiondea @jsquire could you please confirm you can use the repro code to reproduce the issue? Thank you.

[nemakam]

@SeanFeldman , the bug is being worked on. The repro code is doing its job just fine! (y)

[SeanFeldman]

@AlexGhiondea @jsquire you might want to label this issue as a Bug with an optional customer-reported.

[AlexGhiondea]

@binzy @makam can you take a look at this?

[nemakam]

@AlexGhiondea , I'm working on it..

[nemakam]

This was a service side issue which has now been fixed. The code is yet to be rolled out. Thanks for opening this issue.

[SeanFeldman]

Thank you, @nemakam.
Would suggest to keep it opened until it's deployed and rolled out globally. Thanks 🙂

[joshuataylor]

Any ETA for when this will be rolled out? Been seeing this for 7 months, so would be glad to see this resolved.

[nemakam]

@joshuataylor - Hard to give a precise timeframe, but I would say 1 month.

[joshuataylor]

Okay, thanks.

Also, seeing this error under "User errors" in Azure, when in fact this is a server error, if Azure needs to fix it? We've removed a lot of our code base to not rely on ASB as a result of this.

[nemakam]

@joshuataylor - "transactionID was not found" is actually considered user error.
For analogy, It is exactly same as MessageLockLostException. If you try to complete a message after the lock timeout, we would end up throwing lock lost exception. It is classified as user error. But lets say load balancing got triggered and the container hosting your entity is moved. The lock itself is not persisted and hence is lost on the server. When the client tries to complete the message, it would get lockLostException, though it wasn't the client's mistake in this case.
The transactionID error is similar. If you try to send a message with a transactionId which is already expired, you would get this error. In this case, there was a bug where transactions would get aborted prematurely and hence you would get this error.
I hope this makes it clear.

[nemakam]

If you find any error which you think wasn't caused by the client and if its happening more often than it should (assuming that you cannot completely get rid of non internal-server-errors), you could raise a support ticket on the azure portal and the support staff can help you identify the reason.

[SeanFeldman]

@jfggdl any updates on this? Thank you.

[SeanFeldman]

@jfggdl I've tested again and the behaviour is still broken, indicating the fix might not have been deployed yet. Could you please validate if that's the case or not?

I'm assuming my comment can be considered customer-response-expected. If not, what the label stands for? Thank you.

[jfggdl]

@SeanFeldman, Deployment should be finished within a week.
Yes, you are correct on the label Sean. Thanks!

[SeanFeldman]

@jfggdl I've re-run a test and it looks like the fix is in place 🎉
Thank you and the team on my and my customers' behalf.