Azure-sdk-for-net: [FEATURE REQ] Enable application insights end to end tracing for Event Grid

[Expecho]

Library or service name.
Microsoft.Azure.EventGrid

Is your feature request related to a problem? Please describe.
When using the SDK to send events and receive them (using Azure Functions for example) correlation data is not send which results in Application Insights not being able to correlate the event sending with the event handling.

I would like to see something similar as done with the Azure Service Bus.

Currently it is very hard to get end-to-end tracing when Event Grid is involved.

References
https://github.com/microsoft/ApplicationInsights-dotnet/issues/1427
https://feedback.azure.com/forums/909934-azure-event-grid/suggestions/33306310-header-pass-through
https://feedback.azure.com/forums/357324-azure-monitor-application-insights/suggestions/40586104-support-for-correlated-tracing-through-event-grid

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @jfggdl.

[jsquire]

//fyi: @ellismg, @JoshLove-msft

[ahamad-MS]

Copied the same answer from issue #13575

We are adding distributed tracing and OpenTelemetry support for CloudEvent 1.0 in our Data plane SDKs which we are currently working on (No final ETA yet). We will follow the approach documented here for CloudEvent 1.0: https://github.com/cloudevents/spec/blob/master/extensions/distributed-tracing.md

We are still in discussion on how to support that in the SDK for event grid event schema. This will require modifying the current event grid schema to include new properties for tracing. Again, no commitment on this work or ETA yet.

Will you be able to use CloudEvent initially? Does this answer your question/request?

[ahamad-MS]

@Expecho Please feel free to close one of the two issues and keep the other one for tracking since both are about the same support/ask. thanks

[jfggdl]

@Expecho, in a week or so we are going to be releasing a new SDK preview version that provides distributed tracing support. Please note that this support is not offered in the current version of the SDKs (Microsoft.Azure.EventGrid which is the one that you are using), but for the new version currently in preview (Azure.Messaging.EventGrid). Please try it and let us know if you still have unmet requirements. This is the link to Azure SDKs. Just enter Event Grid in the search field: https://azure.github.io/azure-sdk/

[msftbot[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[Expecho]

I suppose I have to wait for an update of the Microsoft.Azure.WebJobs.Extensions.Event package, otherwise my azure function won;t be able to use the new Azure.Messaging.EventGrid package. Are there plans to create a preview of Microsoft.Azure.WebJobs.Extensions.Event that references the new Azure.Messaging.EventGrid package?

And does the new package provide tracing for the event grid event schema as well, or is the CloudEvent schema the only supported schema? Looking at the source I suspect only CloudEvent is supported for the end to end tracing. Correct? If so, the bad thing is that Event Grid triggered Azure Functions only support the Event Grid Event schema:

Currently, you can't use an Event Grid trigger for an Azure Functions app when the event is delivered in the CloudEvents schema.

Of course I could use an http triggered function as outlined here but:

It is strongly recommended that you use an Event Grid Trigger when triggering an Azure Function with Event Grid. The use of a generic WebHook trigger here is demonstrative.

[JoshLove-msft]

I suppose I have to wait for an update of the Microsoft.Azure.WebJobs.Extensions.Event package, otherwise my azure function won;t be able to use the new Azure.Messaging.EventGrid package. Are there plans to create a preview of Microsoft.Azure.WebJobs.Extensions.Event that references the new Azure.Messaging.EventGrid package?

Yes, there are plans to release a preview of the extensions package that references the new EventGrid library. @pakrym may have more details on this.

And does the new package provide tracing for the event grid event schema as well, or is the CloudEvent schema the only supported schema? Looking at the source I suspect only CloudEvent is supported for the end to end tracing. Correct? If so, the bad thing is that Event Grid triggered Azure Functions only support the Event Grid Event schema:

Tracing is supported when publishing both EventGrid events and CloudEvents. However, you are right that only the CloudEvent schema has a place for us to store the tracing information in the event itself to enable end to end scenarios.

We do have an issue created for the work needed to support CloudEvents in the extensions - https://github.com/Azure/azure-sdk-for-net/issues/15466

[JoshLove-msft]

Closing as this is enabled in the new library.

[Expecho]

@pakrym, since this issue is closed and cannot be used to track the outstanding work, do you have an issue number I can subscribe to so I know when I can actually update my solutions to use end-to-end tracing?

[Expecho]

Closing as this is enabled in the new library.

@JoshLove-msft I feel this might be a bit premature. The new library is still in preview, also there is no azure function integration yet we can use. So end to end tracing of Event Grid in conjuction with Azure Function Event Grid trigger isn't really supported yet.

One last question: in https://github.com/Azure/azure-sdk-for-net/issues/13575 the last comment says:

We are still in discussion on how to support that in the SDK for event grid event schema. This will require modifying the current event grid schema to include new properties for tracing. Again, no commitment on this work or ETA yet.

It this still being discussed or is support for event grid event schema definitely dropped for the future? For now I have a workaround here but it requires quit a lot of overhead code.

[pakrym]

@JoshLove-msft let's keep this open to verify the tracking E2E.

[JoshLove-msft]

One last question: in #13575 the last comment says:

We are still in discussion on how to support that in the SDK for event grid event schema. This will require modifying the current event grid schema to include new properties for tracing. Again, no commitment on this work or ETA yet.

It this still being discussed or is support for event grid event schema definitely dropped for the future? For now I have a workaround here but it requires quit a lot of overhead code.

Adding @ahamad-MS to see if he has any updates on this piece.