Apm-agent-dotnet: Upload on blob storage produces "{AzureBlobStorageDiagnosticListener} Failed capturing DiagnosticSource event" error

[NicolasREY69330]

APM Agent version and Azure Blob storage

<PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.11.0" />
<PackageReference Include="Azure.Storage.Blobs" Version="12.9.1" />

Environment

Windows 10 WSL2 => Docker (Ubuntu)

.NET Framework/Core name and version and Application Target Framework(s)
<TargetFramework>net5.0</TargetFramework>

Describe the bug

When uploading a pdf file to Azure Blobstorage, APM is generating a listener error (when calling UploadAsync).

To Reproduce

Steps to reproduce the behavior:

1. Register APM

        public static IApplicationBuilder UseElasticSearchAPM(this IApplicationBuilder app, IConfiguration config)
        {
            if (!Elastic.Apm.Agent.IsConfigured)
                app.UseAllElasticApm(config);

            return app;
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
       // Distributed logs
        app.UseElasticSearchAPM(Configuration);
            Agent.Subscribe(new AzureBlobStorageDiagnosticsSubscriber()); // Add Azure Storage Listener explicitly (https://www.elastic.co/guide/en/apm/agent/dotnet/current/setup-azure-storage.html)

appsettings.json

  "ElasticApm": {
    "ServerUrls": "http://apm:8200",
    "Enabled": true,
    "TransactionSampleRate": 1,
    "CaptureBody": "all",
    "CaptureHeaders": true,
    "SpanFramesMinDuration": 0,
    "CloudProvider": "none"
  },
  "AzureBlobStorage": {
    "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=xxxx;AccountKey=yyyyyyy;EndpointSuffix=core.windows.net",
    "ContainerName": "invoice"
  }

1. Then store a pdf document to Azure Blob Storage

        private BlobContainerClient _containerClient;
        private readonly string _container;

        public AzureBlobStorageRepository(Configuration configuration)
        {
            _container = configuration.ContainerName;
            _containerClient = new BlobContainerClient(configuration.ConnectionString, _container);
        }
        public async Task<Uri> UploadAsync(string friendlyFileName, string path, string blobName, byte[] stream)
        {
            // https://github.com/Azure/azure-sdk-for-net/issues/11770
            await _containerClient.CreateIfNotExistsAsync();

            // ie /2021/06/hash
            var fullfileNamePath = Path.Combine(path, blobName);

            // Get a reference to a blob
            var blob = _containerClient.GetBlobClient(fullfileNamePath);

            // Headers
            var contentDisposition = $"inline; filename={friendlyFileName}";
            var headers = new BlobHttpHeaders
            {
                ContentType = "application/pdf",
                ContentDisposition = contentDisposition
            };

            var uploadedFile = await blob.UploadAsync(new MemoryStream(stream), headers); // produces the provided APM error, even if the method stores the document as expected
            return blob.Uri;
        }

1. See error

[14:34:09 ERR] {AzureBlobStorageDiagnosticListener} Failed capturing DiagnosticSource event
System.ArgumentNullException: Value cannot be null. (Parameter 'uriString')
   at System.Uri..ctor(String uriString)
   at Elastic.Apm.Azure.Storage.BlobUrl..ctor(String url)
   at Elastic.Apm.Azure.Storage.AzureBlobStorageDiagnosticListener.OnStart(KeyValuePair`2 kv, String action)
   at Elastic.Apm.Azure.Storage.AzureBlobStorageDiagnosticListener.HandleOnNext(KeyValuePair`2 kv)
   at Elastic.Apm.DiagnosticListeners.DiagnosticListenerBase.OnNext(KeyValuePair`2 kv)
[14:34:09 ERR] {AzureBlobStorageDiagnosticListener} Failed capturing DiagnosticSource event
System.ArgumentNullException: Value cannot be null. (Parameter 'uriString')
   at System.Uri..ctor(String uriString)
   at Elastic.Apm.Azure.Storage.BlobUrl..ctor(String url)
   at Elastic.Apm.Azure.Storage.AzureBlobStorageDiagnosticListener.OnStart(KeyValuePair`2 kv, String action)
   at Elastic.Apm.Azure.Storage.AzureBlobStorageDiagnosticListener.HandleOnNext(KeyValuePair`2 kv)
   at Elastic.Apm.DiagnosticListeners.DiagnosticListenerBase.OnNext(KeyValuePair`2 kv)

Expected behavior

No error

[russcam]

Hi @NicolasREY69330,

Thanks for reporting. From initial investigation, it appears that the setting of the "url" in the tags of the Activity that is raised by the Azure.Storage.Blobs nuget package when performing an operation, has been removed in Azure.Storage.Blobs version 12.9.0 onwards. I've opened https://github.com/Azure/azure-sdk-for-net/issues/22709 to discuss with the Azure folks. In the meantime, as a workaround, you'd be able to use Azure.Storage.Blobs 12.8.4, and get these traces.

_Sidenote:_
I notice that both app.UseAllElasticApm(config); and Agent.Subscribe(new AzureBlobStorageDiagnosticsSubscriber()); are called; app.UseAllElasticApm(config); will also subscribe AzureBlobStorageDiagnosticsSubscriber so the latter, explicit subscription should be removed.

[NicolasREY69330]

Hi @russcam
Thank you for your feedback, I'll downgrade the Azure blob storage package then, and thanks for the sidenote, I'm going to remove the explicit redundant subscribing operation.

[kipusoep]

We're encountering this as well and look forward to the next release. Any clue about a timeframe?

[russcam]

@kipusoep we would like to get this out in the next release. I've opened #1484 with a fix. Once this PR is merged, we'll discuss plans for a release.