Aspnetcore: EventSource? DiagnosticSource? Both?

[markrendle]

I'm seeing issues opened by @anurse across multiple repos regarding EventSource tracing (like https://github.com/aspnet/Security/issues/1518 for example), but no actual activity as yet; no trace of EventSource anywhere in MVC.

I've been writing a bunch of code to consume DiagnosticSource tracing with the new Activity stuff lately, so I was kinda hoping those same repos would be adding DiagnosticSource support.

Is there a long-term plan to support one over the other? They seem to do the same thing. Obviously on Windows EventSource writes to ETW, whereas on Linux I guess it's in-proc?

Any guidance would be much appreciated.

[Eilon]

@rynowak - do we have guidance on this that we can share?

[rynowak]

A few quick thoughts I've repeated many times internally, but I'm not sure about externally...

EventSource

Assume that any use of EventSource also implies the use of an out of process logging mechanism like LTTNG or ETW. EventSource is great when you want to fire off some data points with low fidelity information to be looked at later, and more likely compiled into some kind of statistics or summary. I say low fidelity because the common usage for EventSource is to pipe the events to a file (ETW) and then use them to do postmortem analysis. Tools that look at data from EventSource like PerfView have special knowledge of some events (JIT, GC) but for the events that you write, they show a generic view. Showing a generic view or summaries/tables of data works because it is all simple types. You can of course build specialized tools, but the common case is to use something like PerfView.

The next statement is going to seem obvious, but my reason for mentioning it will hopefully become clear in a moment. With EventSource the interpretation of the events you fire is determined ahead of time. All of the data you want to record is specified by the component author and it has a fixed meaning, usually associated with the event name. With EventSource you often say things like "read 4096 bytes of data" or "loaded file in 5.6ms". These little statements of fact can be compiled into summaries like "150ms spent doing file i/o" or drilled into specifically if they are outliers.

DiagnosticSource

Assume that any use of DiagnosticSource is being done in-process by a 'plugin' or another logging system that is going to distill your data into meaningful nuggets. DiagnosticSource is great when you want to pass the full-context of the application's state to a listener and that listener is going to add their own interpretation based on access to the application's state. DiagnosticSource also has no native logging format, and no generic understanding of its data points (as they are not usually simple types) - it requires an additional system to create something users want to see.

DiagnosticSource events tend to be chunkier and say things like "I am about to process a request" or "I caught an exception thrown by middleware". These are still statements of fact, but the scope is much larger. Since tools have access to rich context, they are free to capture whatever information is meaningful according to context. Ultimately the set of information that's captured by a tool like Application Insights isn't defined by the ASP.NET team. We provide them all of the context we have and they capture what they think is important.

You can also pipe DiagnosticSource data into EventSource through a bridge. I haven't personally done this, but I seems intended to avoid doubling the amount of diagnostics code necessary in some places. https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/DiagnosticSourceEventSource.cs#L22 I wouldn't assume that you'd use DiagnosticSource in all of the same places as you would EventSource. I'm not sure how a strategy leveraging this would play out, I haven't tried it.

For us, this area has changed a few times. We started here by partnering with Glimpse, we eventually got to partnering with Application Insights. Along the way a few other things have been built on top of Diagnostic Source but I'm not sure what the status of those other tools is right now.

Our Plans

I can't speak to the larger strategy around EventSource or DiagnosticSource - but I can answer how I think of them and how most of the team views our approach. We didn't create DiagnosticSource as a replacements for EventSource. In my mind it serves a different consumer and set of scenarios.

We add diagnostics source events where we think it will add a lot of value. A few of these go a really long way because they are like extensibilitity for someone else to build diagnostics. I'd venture that hosting has maybe 3-5 diagnostic source points and MVC has closer to 10. I'm not sure if there are any places we're currently tracking adding more. If anyone has a genuine need we'd consider it.

We're doing a build up of event source. This is something that our field support engineers are familiar with, and it's generally an important part of perf analysis for teams at Microsoft. We're still trying to solve the void left by the absence of performance counters. The .NET team is working to make EventCounters (built on top of EventSource) as suitable replacements. The main reason for a delay in the ASP.NET team investing more in EventSource is that for a while it wasn't clear what the strategy would be for non-Windows. We wanted to avoid building another additional thing in the future if EventSource wasn't going to be it for *nix platforms. That's been cleared up.

There's a good doc here: https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md#instrumenting-with-diagnosticsourcediagnosticlistener with some thoughts from some folks that are more closely responsible for the strategy in this area.

For You

Well, it sounds like you can do both or either. I'm not very familiar with what you're working on, so what follows is some general advice.

First, think about who your consumer is. Is this for you? (perf) Is this for application developers? (debugging and troubleshooting) Is this for tool authors? Are the tools running postmortem or inside the app?

Secondly, think about how many places you plan to instrument and what kind of data you think is meaningful. How will the consumers understand and interpret this data? Are the individual data points meaningful or should data be aggregated?

[aspnet-hello]

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.