Runtime: What API should I use to consume ETW?

[ghost]

What API souuld I use to consume ETW:

1. System.Diagnostics.Tracing.EventSource (BCL)
2. Microsoft.Diagnostics.Tracing.EventSource (NuGet)
3. Microsoft.Diagnostics.Tracing.TraceEvent (NuGet)
4. krabsetw (NuGet)

?
And where Microsoft.Diagnostics.Tracing.Logging fits in?

I read this at https://blogs.technet.microsoft.com/office365security/hidden-treasure-intrusion-detection-with-etw-part-2/:

TraceEvent is a library used by the PerfView tool and has the benefits of being a well-designed .NET API. Unfortunately, it doesn’t perform well for scenarios where we want to keep memory usage to a minimum. System.Diagnostics.Tracing has the advantage of being part of the .NET BCL but we’ve observed intermittent exceptions and unexpected behavior in the past. Additionally, it suffers from the same memory consumption issue that TraceEvent does.
In response to these challenges, Office 365 Security chose to implement our own API with three primary goals:
•Intuitive and flexible API
•High performance – filtering events in the native layer
•Available both in .NET and native C++
The result of this work is krabsetw, a library we’ve open-sourced under the MIT license. It contains both a native C++ API as well as a .NET API. This library is used in production today across Office 365 workloads on more than 100,000 machines. With filtering, we’re able to process more than more than 500 billion events per day, generating more than 7TB of data per day across the machines.

Is this still true?.. It is written one year ago (May 9, 2017), and other API's have updates since.
I want to know what API I can depend on, if not now, at least in upcoming updates.

Another question: Do I have to use these NuGets too?
Microsoft.Diagnostics.Tracing.TraceEvent.SupportFiles
Microsoft.Diagnostics.Tracing.EventRegister

[stephentoub]

issues in corefx aren't a great place to ask questions like this. Please instead ask on sites like stackoverflow.com. Issues are meant to be for reporting problems, proposing features, etc.

[ghost]

@stephentoub
I disagree. There are already issues marked as questions.
This type of question can't be asked in other places, because its about Microsoft team future intentions:
Are they going to kill one ore more of these APIs and depend totally on another?
Are they going to merge them into one?
Are they going to continue updating each API on its separate path, because each has its unique usage and advantages?
Besides, this question has a polite criticism about such a mess! I got lost between all these APIs, and I want to know what is the best API to spend my time in learning.
Thanks.

[ghost]

So, I also asked it in stackoverflow:
https://stackoverflow.com/questions/49707913/what-api-souuld-i-use-to-consume-etw
I wait your ansewer there :)
Thanks.

[ghost]

@stephentoub
This is the stackoverflow response:

put on hold as off-topic by Nkosi, Machavity, Stephen Kennedy, Hovercraft Full Of Eels, Martin James 27 mins ago
This question appears to be off-topic for this site. While what’s on- and off-topic is not always intuitive, you can learn more about it by reading the help center. The users who voted to close gave this specific reason:
•"Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – Nkosi, Machavity, Stephen Kennedy, Hovercraft Full Of Eels, Martin James
If this question can be reworded to fit the rules in the help center, please edit your question.

As I said before, no one can give a real answer about such a question but Microsoft team!
Thanks.

[danmosemsft]

@brianrob thoughts?

[joshfree]

/cc @vancem

[jnm2]

@MohammadHamdyGhanem So far I've only consumed ETW in code for testing purposes to make sure that no async void methods are unfinished by a certain point in the test.

Once I figured out how to use it, I was happy that I didn't need anything third-party.

If you're interested, you can see what I'm doing with eventListener.

[ghost]

@jnm2
Thaanks, I will look at it and sure I will get more experience.
I was updating my book about. Net framework (now I'm adding hints about. Net core as well) published since 2010 for Arabic readers. I decided to add some pages about Debug abd Trace classes, but I ended up with a compleate chapter about what can be called a tracing framework! The EventProvider class lead me to read about ETW, so I found these 4 APIs and I have to take a decision about which API I should introduce to my readers, and whitch I shouldn't. This decision affects the time I should pay, the volume of the book, and of cource its printing cost.
This is why I needed to ask this question. It dosn't matter what is my actual practical need at this point, becuase practical needs of my readers can vary. It will help me to ignore any API that has no future. If an API currently has some problems and issues but is planned to evolve, I should introduce it to the readers.
I got an answer from krabsetw team, stating it was needed after the other APIs faild to handle more than 10 providers:
https://github.com/Microsoft/krabsetw/issues/44#issuecomment-379477652
So, what about the other three (or say two)?
Which one should prevail? Or are they both needed in some cases?
Thanks.

[ghost]

@stephentoub @danmosemsft @brianrob @vancem
why don't you bring krabsetw advantages to EventSource?
https://github.com/Microsoft/krabsetw/issues/44#issuecomment-379477652

[brianrob]

@MohammadHamdyGhanem let me introduce each of these packages:

System.Diagnostics.Tracing.EventSource: This is the copy of EventSource that lives in the .NET frameworks (full, core, uwp) and can be used by frameworks, libraries, and applications to log events that can be sent to platform loggers (ETW, LTTng) and EventListeners. It is strictly used for producing events and does not consume events.

System.Diagnostics.Tracing.EventListener: This is the copy of EventListener that lives in the .NET frameworks (full, core, uwp) and can be used by frameworks, libraries, and applications to consume events written using EventSources inside the process. They cannot consume events written by other processes.

Microsoft.Diagnostics.Tracing: A copy of the .NET Core version of EventSource and EventListener targeted at the full desktop framework. Available in the NuGet package Microsoft.Diagnostics.Tracing.EventSource.Redist. This package is designed to allow users of the full desktop framework to consume fixes and/or features in EventSource/EventListener that haven't yet made it into the full desktop framework. It is always recommended to use System.Diagnostics.Tracing unless you specifically need something that is only available in this package.

Microsoft.Diagnostics.Tracing.TraceEvent: A control/consumer library for various logging systems that can be used by .NET applications. The level of support for each logging system varies, but there is support for ETW, LTTng, EventPipe (in-process .NET Core logging), as well as some other XML and JSON based formats. Generally, we say that if you need to consume events from the runtime that TraceEvent is a good way to do it.

krabsetw: This is the first time I've seen this one, so I can't really comment on it.

Microsoft.Diagnostics.Tracing.Logging: This is the first time I've seen this one as well. It looks like a library that is layered on top of TraceEvent to control/consume ETW and is specific to the desktop full framework.

So now, to answer your specific question: If you need to consume ETW events in managed code, we recommend using Microsoft.Diagnostics.Tracing.TraceEvent. If you want to write ETW events or write events in managed code that are consumed within the process using EventListeners, we recommend using System.Diagnostics.Tracing.EventSource.

You should not need to use Microsoft.Diagnostics.Tracing.TraceEvent.SupportFiles or
Microsoft.Diagnostics.Tracing.EventRegister. SupportFiles is and internal implementation detail of the build of Microsoft.Diagnostics.Tracing.TraceEvent. EventRegister is used on Windows machines if you need to get and register and an ETW manifest - generally, you don't need to do this.

[ghost]

@brianrob
Thanks a lot for your detailed answer. It helped.

krabsetw: This is the first time I've seen this one, so I can't really comment on it.

I got an answer from krabsetw team, stating it was needed after the other APIs faild to handle more than 10 providers in office 365:
Microsoft/krabsetw#44
I wonder why they didn't report this to you, to talk advantage of their work! At the end you are a one big happy corporation :)
Thanks again.

[brianrob]

Thanks @MohammadHamdyGhanem. I'll follow-up and see if I can find out what the issue was. I'll go ahead and close this issue.

[ghost]

@brianrob
EventLog class reads and writes events to windows logs. I think an ETW class (or any other name) that reads and writes to ETW can make things easier. It can be just a wrapper arround other classes at least to do basic tasks, and who wants to go deeper can use those classes.
Also, I see a problem in naming these namespaces and classses. Most of them contain Trace, Event and Source. We have a Trace class, TraceSource class, EventSource class, TraceEvent class and Eventlog class....etc!

[brianrob]

@MohammadHamdyGhanem, in general, we've tried to move towards a model where you log once and can have multiple consumers. EventSource is the source of such events. And then there are multiple consumers. EventSources can write to EventListeners, ETW (on Windows), LTTng (on Linux), and the EventLog. Furthermore, EventListeners are just programmatic consumers of event sources and can be implemented to send events anywhere.