Azure-sdk-for-net: [BUG] NullReferenceException thrown from Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync when specifying high StorageTransferOptions.MaximumConcurrency

[ohadschn]

Describe the bug
This appears like a regression in the latest version (does not reproduce in 12.2.2):

System.NullReferenceException: Object reference not set to an instance of an object.
   at Azure.Storage.Files.DataLake.DataLakeFileClient.<>c__DisplayClass83_0.<<GetPartitionedUploaderBehaviors>b__2>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Azure.Storage.PartitionedUploader`2.StagePartitionAndDisposeInternal(SlicedStream partition, Int64 offset, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInParallelAsync(Stream content, Nullable`1 contentLength, Int64 blockSize, TServiceSpecificArgs args, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at Azure.Storage.PartitionedUploader`2.UploadInternal(Stream content, TServiceSpecificArgs args, IProgress`1 progressHandler, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.StagedUploadInternal(Stream content, DataLakeFileUploadOptions options, Boolean async, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, DataLakeFileUploadOptions options, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.UploadAsync(String path, PathHttpHeaders httpHeaders, DataLakeRequestConditions conditions, IProgress`1 progressHandler, StorageTransferOptions transferOptions, CancellationToken cancellationToken)

To Reproduce
Steps to reproduce the behavior (include a code snippet, screenshot, or any additional information that might help us reproduce the issue)

await fileClient.UploadAsync(someLargeFileLike2GB, transferOptions: new StorageTransferOptions
            { 
                MaximumConcurrency = 500,
                InitialTransferLength = 1024 * 1024 * 4,
                MaximumTransferLength = 1024 * 1024 * 4
            })

Environment:

• Name and version of the Library package used: Azure.Storage.Files.DataLake 12.3.1
• Hosting platform or OS and .NET runtime version:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.401
 Commit:    5b6f5e5005

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.401\

Host (useful for support):
  Version: 3.1.7
  Commit:  fcfdef8d6b

.NET Core SDKs installed:
  1.1.11 [C:\Program Files\dotnet\sdk]
  1.1.14 [C:\Program Files\dotnet\sdk]
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.503 [C:\Program Files\dotnet\sdk]
  2.1.514 [C:\Program Files\dotnet\sdk]
  2.1.515 [C:\Program Files\dotnet\sdk]
  3.1.302 [C:\Program Files\dotnet\sdk]
  3.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.18 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.19 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.6 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

• IDE and version : Visual Studio 16.7.2

[jsquire]

Thank you for your feedback. Tagging and routing to the team best able to assist.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

[ohadschn]

If it's any help, the null reference is args.Conditions in the UploadPartition behavior:
https://github.com/Azure/azure-sdk-for-net/blob/d89dbc0b9f72f0b8e5f8d53f178eeefc048f944e/sdk/storage/Azure.Storage.Files.DataLake/src/DataLakeFileClient.cs#L4124

When I changed it to args.Conditions?.LeaseId it seemed to work, but I have no idea if that's the proper fix (just needed to execute locally to test something unrelated).

[sumantmehtams]

Adding @seanmcc-msft to take a look.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[seanmcc-msft]

Hi @ohadschn, which upload overload are you using, stream or file path? Can you repo in v12.4.0?

Also, I'd recommend setting MaximumConcurrency to something much lower, maybe like 8. Otherwise you will saturate your network connection, and your uploads will be slower.

[ohadschn]

I think it happened with both overloads, but indeed I can't repro it in 12.4.0.
It looks like you fixed it here: https://github.com/Azure/azure-sdk-for-net/pull/14301 - thanks!

(PS I know about the concurrency recommendations, it was just a simple way to repro)