Azure-sdk-for-java: [BUG] Hitting the exception 'cannot make a new request because the previous response is still open: please call response.close()'

[vashisthsaurabh]

Describe the bug
While using the Azure java SDK to make API calls to Azure, we faced this issue. After making a few number of requests to the list storage accounts API call, we hit the exception _java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()_

Exception or Stack Trace

java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
        at rx.exceptions.Exceptions.propagate(Exceptions.java:57) ~[sd-0.1.jar:?]
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463) ~[sd-0.1.jar:?]
        at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner$61.nextPage(VirtualMachinesInner.java:1749) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.PagedListConverter$2.nextPage(PagedListConverter.java:70) ~[sd-0.1.jar:?]
        at com.microsoft.azure.PagedList.cachePage(PagedList.java:62) ~[sd-0.1.jar:?]
        at com.microsoft.azure.PagedList.<init>(PagedList.java:56) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.PagedListConverter$2.<init>(PagedListConverter.java:67) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.PagedListConverter.convert(PagedListConverter.java:67) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.ReadableWrappersImpl.wrapList(ReadableWrappersImpl.java:43) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.TopLevelModifiableResourcesImpl.list(TopLevelModifiableResourcesImpl.java:118) ~[sd-0.1.jar:?]


Caused by: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
        at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:110) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[sd-0.1.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[sd-0.1.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:186) ~[sd-0.1.jar:?]
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40) ~[sd-0.1.jar:?]
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[sd-0.1.jar:?]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[sd-0.1.jar:?]
        at rx.Observable.subscribe(Observable.java:10423) ~[sd-0.1.jar:?]
        at rx.Observable.subscribe(Observable.java:10390) ~[sd-0.1.jar:?]
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) ~[sd-0.1.jar:?]
        ... 65 more
Caused by: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
        at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:164) ~[sd-0.1.jar:?]
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:108) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:43) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[sd-0.1.jar:?]
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[sd-0.1.jar:?]
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:221) ~[sd-0.1.jar:?]
        at okhttp3.RealCall.execute(RealCall.java:81) ~[sd-0.1.jar:?]
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:186) ~[sd-0.1.jar:?]
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40) ~[sd-0.1.jar:?]
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24) ~[sd-0.1.jar:?]
        at rx.Observable.unsafeSubscribe(Observable.java:10327) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48) ~[sd-0.1.jar:?]
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30) ~[sd-0.1.jar:?]
        at rx.Observable.subscribe(Observable.java:10423) ~[sd-0.1.jar:?]
        at rx.Observable.subscribe(Observable.java:10390) ~[sd-0.1.jar:?]
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443) ~[sd-0.1.jar:?]
        ... 65 more

To Reproduce
Created a azure client and called the list storage accounts API call 100 times using the following snippet:
``` val appCredentials =
new ApplicationTokenCredentials(
clientId,
tenantId,
clientSecret,
azureEnvironment
)

val azure =
Azure.configure.withReadTimeout(timeout,TimeUnit.MILLISECONDS).withConnectionTimeout(timeout,TimeUnit.MILLISECONDS).authenticate(appCredentials).withSubscription("")

for (i <- 1 to 100) {
println(i)
azure.storageAccounts().list()
}

And this fails with the following : 

1
<..>
<..>
<..>
99
2019-11-29T15:53:58.237+0530 ERROR <.main> [compute.AzureComputeClientImpl] An unknown exception is thrown. Will not be retrying.
java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
at rx.exceptions.Exceptions.propagate(Exceptions.java:57) ~[generated-intellij-deps.jar:?]
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463) ~[generated-intellij-deps.jar:?]
at rx.observables.BlockingObservable.single(BlockingObservable.java:340) ~[generated-intellij-deps.jar:?]
at com.microsoft.azure.management.storage.implementation.StorageAccountsInner.list(StorageAccountsInner.java:743) ~[generated-intellij-deps.jar:?]
at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.TopLevelModifiableResourcesImpl.list(TopLevelModifiableResourcesImpl.java:118)
```

Expected behavior
The response should be same everytime and should not give the above exception.

Setup

• Azure java sdk version : 1.24.2
• Azure key vault version : 1.2.2

Additional context
We started seeing this exception when we recently upgraded out okhttp3 package.
Similar issue can be seen here but is closed and should have been resolved with key vault version 1.2.2 : https://github.com/Azure/azure-sdk-for-java/issues/4879

[yaohaizh]

@ChenTanyi, could you help take a look at this issue? Thanks.

[ChenTanyi]

@vashisthsaurabh I cannot reproduce the issue. My code runs with success:

    <dependency>
      <groupId>com.microsoft.azure</groupId>
      <artifactId>azure</artifactId>
      <version>1.24.2</version>
    </dependency>

            ApplicationTokenCredentials credentials = ApplicationTokenCredentials.fromFile(credFile);
            Azure azure = Azure.configure()
//                    .withReadTimeout(1000, TimeUnit.MILLISECONDS)
//                    .withConnectionTimeout(1000, TimeUnit.MILLISECONDS)
                    .authenticate(credentials)
                    .withDefaultSubscription();

            for (int i = 0; i < 200; ++i) {
                azure.storageAccounts().list();
            }

            System.out.println(azure.storageAccounts().list().size()); // 23
// since the timeout will cause a timeout exception. so I run without it.

Can you provide more detail about it? like the timeout number, your storage account number or so on, and could you check whether there are some Failed or Terminated states in your storage accounts?
What do you use for Azure key vault, since the code snippet doesn't need such dependency?

[vashisthsaurabh]

@ChenTanyi Yes sure,

Timeout : 30 mins.
Storage account number : 14

Can this issue appear because of the timeouts or storage account numbers ?

Not sure but I think error in these things should not relate to the above mentioned exception/stack trace. This is somewhere related to the responses not being closed by azure sdk maybe ? And we saw this after we upgraded our okhttp3 version. Can you share the version of okhttp3 you used for this test ?

The code base is quite big and it's not just this snippet. So we might be using Azure key vault in some code path.

[ChenTanyi]

@vashisthsaurabh I just want to double check the parameters. Since the timeout is 30 mins, it is the same as no timeout like the above. So this is mostly your runtime issue about some client didn't close after http response, you could check some other code snippet about it.
My test code is simple, and I didn't directly import okhttp3, my code is: https://github.com/tanyi-test/azure-sdk-for-java-6588

[vashisthsaurabh]

@yaohaizh @ChenTanyi What are the next steps that can help troubleshoot this ?

[ChenTanyi]

@vashisthsaurabh I can't reproduce you problem, thus I am not able to explore further about it. Can you find a code snippet which can reproduce it every time? Thanks.
Like you said above, the may related to somewhere not being closed response, but I don't know your other code. Thus, I cannot know where is the code causing it.

[ashbondu]

@ChenTanyi I feel this may be related to ResourceManagerThrottlingInterceptor

The first retry in ResourceManagerThrottlingInterceptor do not close the response before calling the chain.proceed().

[ChenTanyi]

@ashbondu Do you also meet this problem? If it is related to ResourceManagerThrottlingInterceptor, can you test without it as a try?
change .authenticate(appCredentials) to

        RestClient restClient = new RestClient.Builder()
                .withBaseUrl(credentials.environment(), AzureEnvironment.Endpoint.RESOURCE_MANAGER)
                .withCredentials(credentials)
                .withSerializerAdapter(new JacksonAdapter())
                .withResponseBuilderFactory(new AzureResponseBuilder.Factory())
                .build();

        Azure....authenticate(restClient, credentials.domain())...

[ashbondu]

@ChenTanyi Yes, we have hit this multiple times and it always happens when there are 429s (RetryableError) from the NRP for the vNet operations due to some LB operation in progress.

"properties": {
    "statusCode": "429",
    "serviceRequestId": "23e8c12e-a5d6-4614-bfba-34f7bd593f71",
    "statusMessage": "{\"error\":{\"code\":\"RetryableError\",\"message\":\"A retryable error occurred.\",\"details\":[{\"code\":\"ReferencedResourceNotProvisioned\",\"message\":\"Cannot proceed with operation because resource /subscriptions/<subscriptionId>/resourceGroups/resource/providers/Microsoft.Network/loadBalancers/resource-pod-lb/frontendIPConfigurations/LoadBalancerFrontEnd used by resource /subscriptions/<subscriptionId>/resourceGroups/vnet/providers/Microsoft.Network/virtualNetworks/vnet is not in Succeeded state. Resource is in Deleting state and the last operation that updated/is updating the resource is DeleteLoadBalancerOperation.\"}]}}",
    "responseBody": "{\"error\":{\"code\":\"RetryableError\",\"message\":\"A retryable error occurred.\",\"details\":[{\"code\":\"ReferencedResourceNotProvisioned\",\"message\":\"Cannot proceed with operation because resource /subscriptions/<subscriptionId>/resourceGroups/resource/providers/Microsoft.Network/loadBalancers/resource-pod-lb/frontendIPConfigurations/LoadBalancerFrontEnd used by resource /subscriptions/<subscriptionId>/resourceGroups/vnet/providers/Microsoft.Network/virtualNetworks/vnet is not in Succeeded state. Resource is in Deleting state and the last operation that updated/is updating the resource is DeleteLoadBalancerOperation.\"}]}}"
}

Also want to point out that in one of the systems where we don't use the ResourceManagerThrottlingInterceptor we never run into this issue and we see many 429s there as well.

Client we use is azure-libraries-for-java - v1.26

[ChenTanyi]

@ashbondu Thanks for feedback, we will take a look at it.

[manuelmazzuola]

I've got the same issue, the snippet that trigger the problem

val images = azure
                .virtualMachineImages()
                .listByRegion(REGION)
                .iterator()
images.next()
Thread.sleep(300000) // 300 seconds
images.next() // trigger the exception

that is just a sample, but in the real code we need to sleep because of the API call/hour limit.

[Tony-A-J-Martin]

We are also experiencing this problem in a variety of different situations. We reach the rate limit in different places of our process, for different reasons and instead of retrying correctly, an IllegalStateException is thrown.

An excerpt of the stacktrace:

2020-05-08 10:18:29,655 [WARN ][async-callback-38][com.hybris.cloud.automation.processmanagement.external.ExternalTaskExecutor$ExecutionCallbackImpl][process-id=448811, activiti-id=provision_media_storage, task-id=448828] Failed to execute 'createMediaStorageTask'. Retry it 2020-05-08T10:23:29.655Z. Retries left 5.
java.lang.RuntimeException: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
...
...
...

Caused by: java.io.IOException: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
    at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:110)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.azure.management.resources.fluentcore.utils.ProviderRegistrationInterceptor.intercept(ProviderRegistrationInterceptor.java:40)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.hybris.cloud.automation.azureintegration.AzureLogHeadersInterceptor.intercept(AzureLogHeadersInterceptor.java:34)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
    at okhttp3.RealCall.execute(RealCall.java:81)
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:188)
    at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40)
    at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.DeferredScalarSubscriber.subscribeTo(DeferredScalarSubscriber.java:153)
    at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:32)
    at rx.internal.operators.OnSubscribeTakeLastOne.call(OnSubscribeTakeLastOne.java:22)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.subscribe(Observable.java:10423)
    at rx.Observable.subscribe(Observable.java:10390)
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
    ... 93 common frames omitted
Caused by: java.lang.IllegalStateException: cannot make a new request because the previous response is still open: please call response.close()
    at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:164)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.rest.retry.RetryHandler.intercept(RetryHandler.java:75)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.rest.interceptors.CustomHeadersInterceptor.intercept(CustomHeadersInterceptor.java:140)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.rest.interceptors.UserAgentInterceptor.intercept(UserAgentInterceptor.java:83)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:40)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
    at com.microsoft.azure.management.resources.fluentcore.utils.ResourceManagerThrottlingInterceptor.intercept(ResourceManagerThrottlingInterceptor.java:108)
    ... 132 common frames omitted

[dy10]

+1

[ChenTanyi]

@ashbondu @manuelmazzuola @Tony-A-J-Martin Is there any special environment you are using?
I have tried with 429 and thread sleep, but they all worked as expected. I could not reproduce it. Is there any unclosed in your other code?

It seems many people meet this problem, but if no one providers a code snippet that can reproduce it every time or just by chance, no any further exploration could be done.
(I even tested with multi-thread, but it still cannot reproduce.)
I'll try to close response before sleep, but I'm not sure whether it works.

The screenshot is:

[ChenTanyi]

@vashisthsaurabh @ashbondu @manuelmazzuola @Tony-A-J-Martin @dy10 I've added some response.close in ResourceManagerThrottlingInterceptor, could you have a try with the 1.34.0?

If the exception could be resolved I'll close this issue. Thanks!

[Tony-A-J-Martin]

@ChenTanyi , I can confirm that for us, 1.34.0 resolves the response.close() issue. Now, instead of a RuntimeException, we get a CloudException and the retries are successfully carried out eventually resulting in a successful process. 🥳

Thank you for you help @ChenTanyi , it is very much appreciated.

[ChenTanyi]

@Tony-A-J-Martin Thanks for your confirm.

The issue will be open for a few days and then be closed.