Azure-functions-durable-extension: Distributed tracing doesn't correlate Traces

[NickSevens]

Description

I've setup distributed tracing for Application Insights in my durable functions.
The DtOrchestrator and DtActivity items are correlated correctly, but my Traces (LogInformation, LogDebug, LogTrace) are not correlated to the Activities anymore.
I'm pretty sure this worked a while ago, and also in the early preview NuGet package.
The traces _are_ logged to AI, just not linked to any DtActivity or DtOrchestrator instance.

Expected behavior

I expect to see the Traces correlated to a specific DtActivity, so I can filter them by Activity.
In this pane for example:

Actual behavior

The Trace messages are not correlated to the Activity, so I have to search for them separately to browse logs.

Relevant source code snippets

Logging setup has been tried with and without W3CDistributedTracing enabled
e.g.:

{
  "version": "2.0",
  "logging": {
    "logLevel": {
      "Default": "Information",
      "DurableTask": "Warning",
      "Function": "Trace",
      "Host": "Warning",
      "Microsoft.Azure.WebJobs": "Warning"
    },
    "applicationInsights": {
      "httpAutoCollectionOptions": {
        "enableW3CDistributedTracing": true
      },
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    }
  },
  "extensions": {
    "durableTask": {
      "hubName": "WorkspacesHub",
      "tracing": {
        "distributedTracingEnabled": true,
        "distributedTracingProtocol": "W3CTraceContext"
      }
    }
  }
}

Known workarounds

(None)

App Details

• Durable Functions extension version: v2.4.0
• Azure Functions runtime version: v3
• Programming language used: C#

Screenshots

As an example, these are the properties returned for a DtActivity and should-be-correlated Trace message

[davidmrdavid]

Hi @NickSevens, thank you for bringing this to our attention!

I'm sorry this isn't working as expected, the feature is still in early development and was only recently merged into the main package release, so it's definitely possible some older features stopped working. I'll take a look at this.

In the meantime, it would help me greatly if you could provide a minimal reproducer that illustrates how you're generating these traces and that I could use to determine if your issue has been fixed. Any chance you could provide that? _Thanks!_

⚡ ⚡

[NickSevens]

Hi @davidmrdavid, I tried creating a minimal sample, but there it _does_ correlate correctly. So ... I'm now trying to figure out what's different in my app. :)
I'll post here if I find anything causing the issue at my end...

[mpaul31]

i also noticed the operation name is <Empty> in the portal now after the upgrade so unfortunately this feature is not mature enough for our production workloads quiet yet.

[davidmrdavid]

Hi @mpaul31, you're correct. The feature is still in public preview so the experience needs some more work before being ready for production. Just for my reference, where are you seeing this <Empty> operation name. Do you mind clarifying / sharing a screenshot _Thanks!_

[mpaul31]

Sure. You can see it from the performance blade:

[davidmrdavid]

@mpaul31 , thanks. I'll turn that into a separate ToDo item and prioritize it for an upcoming release of Dist. Tracing

[NickSevens]

@davidmrdavid I've narrowed it down to this:
I recently upgraded a NuGet package which has a dependency on System.Diagnostics.DiagnosticSource. The update caused the dependency to go from 4.6.0 to 4.7.1. This causes the correlation to fail.

When the correlation fails, I noticed a difference in Operation ID format:

• _Working_ correlation Operation ID has a format of for example 00-dc5421953c0c76468418424c05b840ce-70ae1d9822e11c44-00
• _Failing_ correlation Operation ID has a format of for example a61c9fdd-4739b89b1bfe50b0

As a reproduction, you can simply create a new project, and add the NuGet package.
<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="4.6.0" /> --> works
<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="4.7.1" /> --> fails

[davidmrdavid]

Thanks @NickSevens!. I'll need to dive into the documentation to understand why this would break our expected behavior. Thanks for providing this context, this is _really_ helpful :)

[NickSevens]

Not sure if it's the same, but https://tsuyoshiushio.medium.com/system-diagnostics-activity-update-and-distributed-tracing-in-c-f02428dcd050 explains something similar as well.

[lindgrend]

Having the same issue, also if you in the c# sample use Microsoft.Azure.WebJobs.Extensions.DurableTask Version="2.4.0" it also breaks for me. Maybe this can be used as your minimum repro?