azure-sdk-for-net 🚀 - [BUG] .Net Framework hanging on BlobClient.DownloadAsync

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

msftbot[bot] on 22 Apr 2020

@kasobol-msft lets talk tomorrow about this and see if we need to load balance.

amishra-dev on 8 May 2020

@BenKoth I wasn't able to reproduce given provided sample. Could you please run it on your side with a verbose logging (per this page), i.e. add this early in Main method:

            using AzureEventSourceListener listener = new AzureEventSourceListener((e, message) =>
            {
                // Only log messages from Azure-Core event source
                if (e.EventSource.Name == "Azure-Core")
                {
                    File.AppendAllText("C:\\log.txt", $"{DateTime.Now} {message}");
                }
            },
             level: EventLevel.Verbose);

This will generate C:\log.txt file. Please share it with us.

kasobol-msft on 11 May 2020

I was also able to reproduce this behavior using .NET framework 4.6.1 as well and I got a somewhat similar stack trace. Looks like there's some sort of error in Azure.Core

As confirmed this error does not occur in Net Core 2.1 or Core 3.1

System.AggregateException
  HResult=0x80131500
  Message=Retry failed after 6 tries.
  Source=Azure.Core
  StackTrace:
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<ProcessAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.BlobRestClient.Blob.<DownloadAsync>d__0.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.<StartDownloadAsync>d__45.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.<DownloadInternal>d__44.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.<DownloadAsync>d__43.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at StorageTester.Program.<Main>d__0.MoveNext() in C:\Users\amnguye\source\repos\TestOR\TestOR\Program.cs:line 38
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at StorageTester.Program.<Main>(String[] args)

  This exception was originally thrown at this call stack:
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
    System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(System.Threading.Tasks.Task<System.Net.Http.HttpResponseMessage>, System.Net.Http.HttpRequestMessage, System.Threading.CancellationTokenSource, bool)
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
    Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(Azure.Core.HttpMessage)
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
    ...
    [Call Stack Truncated]

Inner Exception 1:
TaskCanceledException: A task was canceled.

Will update the issue once more investigation is done.

amnguye on 12 May 2020

When attempting to enable logging, the error does not occur and error does not occur as often when attempting to debug the storage blob package.

When debugging, it succeeds making 3-4 download calls, but eventually will stall and throw the exception. Fiddler trace shows that the requests were sent, but the method shows it still has not returned yet. Sometimes after a stall, the method will return but not always. Usually the next download call will throw the same exception.

During these stalls the Diagnostic window shows multiple threads being exited with exit code 0.

More investigation may need to be done in Azure.Core to why this is only occurring in .NET Framework 4.6.1/4.7.2

amnguye on 12 May 2020

@pakrym @tg-msft Could we get a bit of help with why Azure.Core is causing this to happen in .NET Framework 4.6.1/4.7.2 ?

amnguye on 14 May 2020

Here is my log file
log.txt

BenKoth on 14 May 2020

https://github.com/Azure/azure-sdk-for-net/pull/12103 might fix this.

pakrym on 15 May 2020

👍1

@pakrym I think this might be it. I've checked server logs, the requests in last retry never made it to the backend. ServicePointManager allows 2 connections by default, so if we were leaking them then they'd be quickly exhausted.

kasobol-msft on 15 May 2020

Hi @pakrym, we have experienced the same issue in our production service. (tag @bruckGT)

When will the this fix be released? In which version of the SDK?
Is there any workaround?

Thanks!