Azure-functions-host: Pass App Insights correlation details to language workers

[brettsam]

In v1, it was much easier to correlate from other langauges, as the Application Insights OperationId was always equal to the invocation id. This isn't the case in v2, due to all of the auto-tracking and cross-site correlation that's now happening. We need to have a way to pass these details down via the context into the languages so that customers can do their own correlation with AppInsights. For example, if they're logging custom metrics or events, they'll want them associated with the current operation.

/cc: @lmolkova -- I'll get your advice here when I go to implement it. I want to make sure we're passing all the relevant details.

[lmolkova]

this question is related to how we want to instrument non-.net functions.
In python and golang our recommendation is opencensus - we don't (and not going to) have officially support ApplicationInsights SDKs for Go/Python.
In node.js and Java AppInsights SDK offerings are more mature than opencensus now, and we are still figuring out what we'd recommend customers to do and transition plan.

The format of the trace context (that carries ids) is different for AppInsights SDKs and opencensus.
So this is kind of broken experience now - if host pass something to the function worker, it may not be something that function expects.

Is there some layer on the function host that can restore this context into something that AI SDK or opencensus can understand?

[peterwain]

We're in the middle of our first Functions v2 project, and have just bumped into this issue.

Our previous projects have been based in v1 and we've become accustomed to seeing all the telemetry relevant to a request being nicely arranged together within the App Insights End-to-end transaction view. As this issue outlines, this is not currently the behavior under v2, and we end up with two clusterings of telemetry per request; one set from the function host under an opaque OperationId, and then our custom telemetry (events, dependencies and metrics) under an OperationId that is derived from the InvocationId.

Are there any current workarounds you can suggest for C#-based functions?

I can see you've done something special with an extension method for logging metrics on ILogger, and that the produced Metrics telemetry inherits the OperationId of the overall request. Could this be extended to the other telemetry types?

[lmolkova]

@peterwain

in V2 it's no longer required to manually set correlation ids on the telemetry.
Check out this example:

https://github.com/brettsam/azure-docs/blob/b574b4bd888c28a826e8442d9ecd176fd99674e0/articles/azure-functions/functions-monitoring.md#functions-version-2-2

As long as you use TelemetryConfigration.Active to send custom telemetry or create a configuration with TelemetryConfiguration.CreateDefault() you get automatic correlation for all kinds of telemetry.

[peterwain]

Thanks so much @lmolkova - that worked perfectly!

I wish I had seen this revised documentation earlier as it would have saved me some headaches. When do you think the currently published documentation will be refreshed with this revised version? EG:
https://docs.microsoft.com/en-us/azure/azure-functions/functions-monitoring

And where's the best place to see the latest documentation for the currently deployed Azure Functions v2?

[lmolkova]

@peterwain I did not realize it's not merged into the official docs.

@brettsam I thought I saw a PR for this change: https://github.com/brettsam/azure-docs/blob/b574b4bd888c28a826e8442d9ecd176fd99674e0/articles/azure-functions/functions-monitoring.md#functions-version-2-2. Do you know what happened?

[irakliy81]

@lmolkova - is there a similar solution for Node.js functions?

[lmolkova]

@irakliy81 unfortunately, no, there is no solution for node.js 👎

[irakliy81]

@lmolkova - thanks for the reply! Do you know if there are any plans to change this?

[andrei-markeev]

One addition to this: based on my testing, at least node.js functions don't even correlate with "x-ms-request-id" or "request-id" header provided from client. So in node.js we don't have any possibility to link e.g. custom events or HTTP dependencies with the request.

[HShather]

Is there plan to fix this for javascript v2 functions?

this issue makes diagnosing errors and performance issues very difficult for us.

this issue, together with 2447 is causing us to think about migrating some of mission critical applications away from azure functions to a web app/web job.

[asselin]

Migrating from v1 to v2 functions to be able to upgrade Node.js to something that won't be out of service next month, and we ran into this same problem-- our custom AI events are no longer correlated to the Request.

Without the correlation, AI becomes FAR less useful.

What's the ETA to have this bug fixed?

Thanks!

[amd032]

Hey guys,
at least two people asked about the progress on this issue, and there's no answer. Does anyone treat the users seriously here? We need this telemetry to work in node, javascript and java, and would like to know when we get it from azure developers.

[fabiocav]

@brettsam / @yojagad moving this back to triage so we can close this milestone. Let's discuss this item today.

[fabiocav]

Closing this as completed.

[pgraham]

This has been closed but I am not finding documentation around how to get access to the operationId for custom telemetry with a node function. The documented solution of using context.invocationId is giving a different ID than the operation ID for host tracked telemetry.

[brettsam]

@yojagad / @mhoeger do we know if https://github.com/Azure/azure-functions-nodejs-worker/pull/244 has made it to production yet?

[mhoeger]

Yeah, it should be a part of 2.0.12888! I don't think docs have been updated though, cc: @yojagad