Azure-sdk-for-java: ServiceBus deadlocks when receiving the messages

[pashashiz]

Describe the bug
There are multiple situations when we might get into a deadlock while receiving messages from ServiceBus. That is critical cause makes it almost impossible to use ServiceBus reliably if we want to control message acknowledgment.

To Reproduce
There are multiple cases, in all cases, we want to acknowledge the message manually.

1. If and the message processing time is larger than the message lock duration set for the subscription and maxAutoLockRenewDuration is Duration.ZERO, the message will be failed to be completed which is expected, but we also expect that the message will be redelivered, but the service bus just hangs and no new messages will be received.
2. If and the message processing time is larger than the message lock duration set for the subscription and maxAutoLockRenewDuration is set to a large value, the message will be competed which is expected, but the service bus just hangs and no new messages will be received.
3. If the message is not acknowledged at all, we expect that message will be redelivered, but the service bus will just hand forever and no new messages will be processed.

Code Snippet
There is an example for case number 2, but that easy to reproduce the rest.

The message lock duration should be set to 10 seconds, in the sample the time to process the message will be 12 seconds. We expect that message will be redelivered over and over, but the consumer will just hang after the first try.

ServiceBusReceiverAsyncClient receiver = new ServiceBusClientBuilder()
        .connectionString(connectionString)
        .receiver()
        .disableAutoComplete()
        .prefetchCount(1)
        .maxAutoLockRenewDuration(Duration.ZERO)
        .topicName("test")
        .subscriptionName("test")
        .buildAsyncClient();

    Disposable subscription = receiver.receiveMessages()
        .flatMap(message -> {
          System.out.printf("Sequence #: %s. Contents: %s%n", message.getSequenceNumber(), message.getBody());
          try {
            Thread.sleep(12_000);
          } catch (InterruptedException e) {
            e.printStackTrace();
          }
          return receiver
              .complete(message)
              .onErrorResume(e -> {
                e.printStackTrace();
                return Mono.empty();
              });
        })
        .doOnError(e -> System.out.println("ERROR: " + e.getMessage()))
        .doOnComplete(() -> System.out.println("DONE"))
        .subscribe();

Expected behavior
In all the cases ServiceBus should not hang and should proceed consuming new messages or redeliver late messages.

Setup (please complete the following information):

• OS: macOS
• IDE : IntelliJ
• Version of the Library used 7.0.1

[hemanttanwar]

Thank You @pashashiz . I am looking into this and will update you on this.

[ktomazin]

I am seeing this exact same issue, except I am using a queue and have disabled auto complete. My instantiation looks like this:
ServiceBusProcessorClient processorClient = createServiceBusClientBuilder()
.connectionString(connectionString)
.processor()
.queueName(queueName)
.disableAutoComplete()
.receiveMode(ServiceBusReceiveMode.PEEK_LOCK)
.processMessage(messageProcessor)
.processError(errorProcessor)
.buildProcessorClient();

Do you have an ETA of when it will be fixed?

[mishuuuuu]

We are having the same issue. For now we are closing and starting the processor onError and it seams to work, but it is an ugly fix to destroy the link like this.

[ktomazin]

Exactly the same thing I just implemented. It is ugly, but it works while
we wait for the real fix.

On Mon, Feb 22, 2021 at 12:17 PM mishuuuuu [e-mail] wrote:

We are having the same issue. For now we are closing and starting the
processor onError and it seams to work, but it is an ugly fix to destroy
the link like this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/19247#issuecomment-783531735,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSC4CMBYBSXOVV644WHHDTAKGRNANCNFSM4XUG3VYA
.

[hemanttanwar]

@pashashiz I have done some testing for case1, case 2 scenario and confirm that this issue exists. I am working on solution. We want to make sure it is tested thoroughly. I am going to test more with case 3 scenario you explained.

[ktomazin]

Thanks so much for the update; looking forward to receiving the fix.

Can you provide an estimate of when it might be available?

Ken

On Tue, Feb 23, 2021 at 8:32 PM Hemant Tanwar [e-mail]
wrote:

@pashashiz https://github.com/pashashiz I have done some testing for
case1, case 2 scenario and confirm that this issue exists. I am working on
solution. We want to make sure it is tested thoroughly. I am going to test
more with case 3 scenario you explained.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/19247#issuecomment-784667083,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSC4A6D4MJZ5XBAFNEPH3TARJMPANCNFSM4XUG3VYA
.

[hemanttanwar]

@pashashiz Regarding case 3 : I like to understand little more.
Why you are not settling the message by any of the settlement API ( i.e. .complete()/abandon() etc) ?

You are using disableAutoComplete (that means you want to settle messages yourself) and message is read in PEEK_LOCK mode (that is default mode), that means message is locked for a fixed duration.
The purpose is for client to process the message and settle after processing.

If you intentionally do not settle it in PEEK_LOCK mode, your client will receive the message and lock will expire after fixed time, message goes back to Service bus and it will increase Delivery count. If this loop repeat and after MaxDelivery count , your message will go in dead letter queue.

If and as you settle the messages, we retrieve more messages from service bus and you will keep getting messages.

[pashashiz]

@hemanttanwar We use the service bus wrapped in Akka streams, imagine you have a large computation graph and an error happens somewhere, right now, we are forced to be very careful and handle every possible case so we 100% either explicitly complete or abandon. Note, that for abandon you have to have a link to the original message, which requires special treatment, catching the regular exception is not enough.

We are afraid if we ever do a small mistake and will not explicitly abandon the message we will get into a deadlock (which already happened few times in production). The expected result is just redelivery like it works in any other messaging system I ever worked with. If the message is redelivered MaxDelivery times that will go into a dead latter queue, that is expected. But freezing an entire service is not what we expect.

[mishuuuuu]

Exactly the same thing I just implemented. It is ugly, but it works while we wait for the real fix.
…
On Mon, Feb 22, 2021 at 12:17 PM mishuuuuu @.*> wrote: We are having the same issue. For now we are closing and starting the processor onError and it seams to work, but it is an ugly fix to destroy the link like this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19247 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTSC4CMBYBSXOVV644WHHDTAKGRNANCNFSM4XUG3VYA .

We had another deadlock with this fix in place. Had it worked ok for you?

[ktomazin]

It seemed to work in my local environment but now that I have it deployed to our dev server I'm seeing intermittent failures like either the messages are never sent or they are sent but never picked up. I'm still scratching my head and trying to figure out what is going on. I'll let you know what I find. Can you provide more details on the deadlock you are seeing? Maybe it will help me figure out what is going on with our system.

[hemanttanwar]

@pashashiz I am planning to release a fix for three original scenario reported in this issue in March month. It should come out in next 2 week or before.

[hemanttanwar]

@pashashiz Please try this version https://repo1.maven.org/maven2/com/azure/azure-messaging-servicebus/7.1.0/

[ktomazin]

Thanks, I'll check it out.

On Thu, Mar 11, 2021 at 12:30 AM Hemant Tanwar @.*>
wrote:

@pashashiz https://github.com/pashashiz Please try this version
https://repo1.maven.org/maven2/com/azure/azure-messaging-servicebus/7.1.0/

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/19247#issuecomment-796466815,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSC4FZBNU2C6I4Z5I6KI3TDBBNDANCNFSM4XUG3VYA
.

[ktomazin]

Hi There,

I have updated my pom file to reference the updated
azure-messaging-servicebus 7.1.0 and now my application is hanging on
startup. Based on the thread dump, I believe it is hanging attempting to
get the shared secrets from the key vault as we are also
using azure-spring-boot-starter-keyvault-secrets:

    <dependency>
        <groupId>com.azure.spring</groupId>

azure-spring-boot-starter-keyvault-secrets
3.0.0-beta.1

I have attempted to upgrade this dependency to the most recent version
(3.2.0) but that didn't seem to help.

I'm attaching the thread dump to this email. Please let me know if there
is anything else we can provide to help you isolate this issue.

Regards,

Ken

On Thu, Mar 11, 2021 at 11:01 AM Ken Tomazin @.*> wrote:

Thanks, I'll check it out.

On Thu, Mar 11, 2021 at 12:30 AM Hemant Tanwar @.*>
wrote:

@pashashiz https://github.com/pashashiz Please try this version
https://repo1.maven.org/maven2/com/azure/azure-messaging-servicebus/7.1.0/

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Azure/azure-sdk-for-java/issues/19247#issuecomment-796466815,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTSC4FZBNU2C6I4Z5I6KI3TDBBNDANCNFSM4XUG3VYA
.

@.*" prio=5 tid=0x1 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
at reactor.core.publisher.Flux.blockLast(Flux.java:2497)
at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94)
- locked <0x118d> (a com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator)
at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.(ContinuablePagedByPageIterable.java:49)
at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:37)
at java.lang.Iterable.spliterator(Iterable.java:101)
at com.azure.spring.keyvault.KeyVaultOperation.lambda$refreshProperties$2(KeyVaultOperation.java:142)
at com.azure.spring.keyvault.KeyVaultOperation$$Lambda$192.1365790282.apply(Unknown Source:-1)
at java.util.Optional.map(Optional.java:215)
at com.azure.spring.keyvault.KeyVaultOperation.refreshProperties(KeyVaultOperation.java:142)
at com.azure.spring.keyvault.KeyVaultOperation.(KeyVaultOperation.java:80)
at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:101)
at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:200)
at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:188)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at com.kroger.bis.DropShipGWApplication.main(DropShipGWApplication.java:15)

@.*" daemon prio=5 tid=0x1a nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x19 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x18 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x17 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x16 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x15 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x14 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=5 tid=0x13 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at io.netty.channel.kqueue.Native.keventWait(Native.java:-1)
at io.netty.channel.kqueue.Native.keventWait(Native.java:94)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:177)
at io.netty.channel.kqueue.KQueueEventLoop.kqueueWait(KQueueEventLoop.java:169)
at io.netty.channel.kqueue.KQueueEventLoop.run(KQueueEventLoop.java:238)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

"RMI @.*" daemon prio=5 tid=0x11 nid=NA waiting
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"RMI TCP @.*" daemon prio=5 tid=0x22 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(SocketInputStream.java:-1)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x1189> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:555)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$154.867358284.run(Unknown Source:-1)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"RMI TCP @.*" daemon prio=5 tid=0x10 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(SocketInputStream.java:-1)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x118a> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:555)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$154.867358284.run(Unknown Source:-1)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

"RMI TCP @.*" daemon prio=5 tid=0xe nid=NA runnable
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:-1)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
at java.lang.Thread.run(Thread.java:748)

@.*" daemon prio=8 tid=0x3 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Object.java:-1)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference @.*" daemon prio=10 tid=0x2 nid=NA waiting
java.lang.Thread.State: WAITING
at java.lang.Object.wait(Object.java:-1)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"Attach @.*" daemon prio=9 tid=0xf nid=NA runnable
java.lang.Thread.State: RUNNABLE

"Signal @.*" daemon prio=9 tid=0x4 nid=NA runnable
java.lang.Thread.State: RUNNABLE

[pashashiz]

@pashashiz Please try this version https://repo1.maven.org/maven2/com/azure/azure-messaging-servicebus/7.1.0/

Looks like everything was fixed. Thanks!!!

[hemanttanwar]

@pashashiz Appreciate you using our library.