Azure-sdk-for-net: [Service Bus] message receiver - complete must be done via the same instance that received the message. Why?

[VaclavK]

Hi,

we are aware of the requirement that message received by the instance of message receiver must be completed by the very same instance. We are trying to understand why that is? Can you please provide any pointers?

Our application runs on Service Fabric and the particular service that is responsible for receiving and completing messages is implemented as a reliable service.. Further, when the message is received by the service it is delegated to another service for dispatch and the service has an endpoint listener to receive callback, which would complete the message.

Given cluster/app lifecycle, what may happen is that the primary that has received the message may be re-delegated/stopped and the message will not succeed to be completed by "new" primary.

We also see receiver stopping receiving during load tests (with low inflight message count) (seems to be similar to https://github.com/Azure/azure-service-bus-dotnet/issues/470)

One pattern we are looking at is that when we see SF API call to service for lifecycle event to allow for remaining complete message callback to be received and handled. But we are hoping to understand this tie in to specific instance of message receiver . Any pointers are appreciated.

update would underlying protocol affect this? If I switch from AMQP to NetMessaging (obsolete), does this message receiver limitation still holds?

[VaclavK]

@adrianhall why the question contains reference to Service fabric, the underlying issue is really about Service Bus message receiver so not sure re those labels

[adrianhall]

Thanks for the clarification.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jfggdl

[VaclavK]

@nemakam

related question - I tried to implement resetting message receiver as per your comment in the linked github ticket

however when the original message receiver owns the connection (that I was passing to subsequent one) it gets disposed when the original owner receiver is disposed so what I do now is
I create shared connection as in

conn = new ServiceBusConnection(new ServiceBusConnectionStringBuilder(serviceBusConnectionString));
..
and then every receiver instance as in

return new MessageReceiver(
conn,
EntityNameHelper.FormatSubscriptionPath(topicName, subscriptionName),
ReceiveMode.PeekLock,
new RetryExponential(
BackoffMin,
BackoffMax,
ReceiverBatchSize),
ReceiverBatchSize);

however such receiver never receives anything... constantly returns null despite topic having

Azure SB package @ 4.0.0

is this approach not what you suggested there? anything wrong you can see with this approach?

[jfggdl]

@Vaclavk, thanks for creating this issue. We acknowledge your scenario and needs. We are going to create a backlog item and we will prioritize and determine if we implement it.

[nemakam]

@VaclavK ,
Yes. As you mentioned, this is by-design and works this way since this library works on top of AMQP protocol. Changing it to NetMessaging should fix your problem.

Now coming to why this is this way -
AMQP is a stateful communication protocol wherein you have links which remember which messages have been delivered from source to destination. While completing the message, it translates to AMQP-Disposition which essentially looks up its own state to see if it was delivered using that link, and then performs a disposition action on that.
We understand that this is a limitation now for the new ServiceBus SDK, but that's how the protocol works.

There are ways to get it working using something we call as Request-Response link pattern, but is something that needs to be implemented on the service side before the client can expose it.
For that we need to gauge if there are others who are interested in this, and then prioritize based on that.

[nemakam]

is this approach not what you suggested there? anything wrong you can see with this approach?

@VaclavK - Is that post related to this issue or is it a different issue? I am not sure I quite understand your issue. Could you explain what is happening and what is it that you are trying to do again?
And if it is not related to the original post, please create a new GitHub issue.

[dguerin]

@VaclavK ,
Yes. As you mentioned, this is by-design and works this way since this library works on top of AMQP protocol. Changing it to NetMessaging should fix your problem.

Now coming to why this is this way -
AMQP is a stateful communication protocol wherein you have links which remember which messages have been delivered from source to destination. While completing the message, it translates to AMQP-Disposition which essentially looks up its own state to see if it was delivered using that link, and then performs a disposition action on that.
We understand that this is a limitation now for the new ServiceBus SDK, but that's how the protocol works.

There are ways to get it working using something we call as Request-Response link pattern, but is something that needs to be implemented on the service side before the client can expose it.
For that we need to gauge if there are others who are interested in this, and then prioritize based on that.

We would be very interesting in this. Particularly in a receiver failover/graceful upgrade flow to prevent us from having to process successfully delivery messages more than once.

[msftbot[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[VaclavK]

I am not sure what more is needed from me.. I believe the issue is understood and is a question of priority... Is there any aspirational ETA at the moment?

[dguerin]

@jsquire if there is anything we can help with here do let us know.

[jsquire]

@dguerin - Thanks for your willingness to assist. Unfortunately, from what I understand, this looks to be blocked on a support from the service-side as a prerequisite to any client enhancements. @nemakam and @axisc would be the folks that help drive that decision and prioritization. They may have suggestions on the best place to raise awareness and gauge community interest.

At this point, I've shifted this to the service backlog for consideration.

[jdimmerman]

This would also service a way to work around issues such as https://github.com/Azure/azure-functions-servicebus-extension/issues/95. Right now, there doesn't appear to be another serviceable workaround other than handling processing messages multiple times.