Benchmarkdotnet: Track Native Memory Allocations and more informations with our ETW Memory Diagnoser

[adamsitnik]

We have two Memory Diagnosers. One of them is using ETW and is deprecated because it's less accurate than the other one. But this diagnoser could actually do more than just tracking allocated managed memory and GC collections.

This issue is up-for-grabs. The person that grabs it should:

• do an ETW research and answer following questions:
  
  
  
  • is it possible to track native memory allocations with ETW? I know that dotTrace can do this, many people asked for it including @JimBobSquarePants
  
  
  • is it possible to track the types of the allocated managed objects? (I know it is, some people asked for it)
  
  
  • is there anything more that is worth of tracking and showing to the users?
  
  
• actual implementation based on extending existing ETW MemoryDiagnoser
• testing, comparing the results with some 3rd party profiler like dotTrace

[goldshtn]

You can track native memory allocations with ETW. However, it would probably slow down your benchmark by introducing ETW events for each allocation and free. You can also track each .NET memory allocation using ETW events. And that would probably slow down your benchmark even more.

TBQH, I don't think this belongs in a benchmarking library. You can profile for allocations while running the benchmark using existing, proven tools.

[adamsitnik]

And that would probably slow down your benchmark even more.

You are 100% right. This is why we perform separate run if any diagnoser is present. In that case we just use the diagnoser's output and ignore benchmark results (which could get affected).

I don't think this belongs in a benchmarking library. You can profile for allocations while running the benchmark using existing, proven tools.

Yes, but on the other hand using our existing ETW Memory Diagnoser to extend it with such features should be relatively easy now. The existing code base is very good, just sign up for a few events and do the math. I believe that with relatively low effort we can add really nice feature.

I see few use cases:

• I know that my code allocates, but I would like to find out what objects are allocated without installing/buying any extra tool.
• I have some weird results, the benchmark should not allocate managed memory, but shows that something is allocated. What is that?
• The same like above but 2 different memory profilers show 0 allocations. I need to check if BenchmarkDotNet is correct or not. (as the maintainer, to possibly find a bug in our Engine which should never allocate)
• I compare few different solutions, one should allocate memory, but shows 0. Maybe it's allocating native memory? (like System.Drawing).

Let me turn on this feature, run the benchmarks and find out!

[goldshtn]

You are basically trying to turn a benchmarking library into a profiler. Where does this stop? For example, it is usually meaningless to just know that I am allocating strings; I want to know where they are coming from, i.e. stacks. Would you want the diagnoser to aggregate and resolve allocating stacks, too? Wouldn't it make more sense to use a profiler if you see suspicious results? This also applies to pure CPU running time: in theory, you could also use ETW to show the hottest methods executed by the benchmark, but you'd probably use a profiler for this, no?

[mattwarren]

You are basically trying to turn a benchmarking library into a profiler.

Just to add a contrary point, JHM does things like this, in fact if even integrates with perf to give you assembly level CPU samples, see JMH perfasm explained for a bit more info. By doing this is makes things like this possible:

• https://shipilev.net/jvm-anatomy-park/6-new-object-stages/
• https://shipilev.net/jvm-anatomy-park/7-initialization-costs/

(not saying we have to copy JHM feature-for-feature, but it's a pretty successful Benchmarking tool, so looking at what it does and being influenced by it isn't a bad idea)

So I think that there is some benefit in having all this integrated into a Benchmarking tool, I'd love to see similar functionality in BenchmarkDotNet.

But I do agree there has to be a limit, some things are better left to profiles.

[goldshtn]

You can get the same results by running perf directly on the benchmarked binary. The development effort of pitting these features in JMH was non-trivial. Of course if you have all the time in the world then you can gradually add features from other tools into your benchmarking library, but generally it is not a good strategy. There's hardly a point in partially replicating features that are available in other free tools, which also includen visualization capabilities that you will probably not add (like PerfView or flame graphs).

[adamsitnik]

@goldshtn I got your point and I am really happy that you are sharing your valuable opinion with us!

I understand that from the business perspective there is no reason for duplicating features that other tools provide.

But "business perspective" affects me from 8 am to 4 pm. After this, I want to do some crazy stuff that solves interesting, hard problems.

My questions is: do you believe that we should use our free time to develop some other features that are not available today? If so, could you provide us some ideas?

I would like to use my free time to create the best benchmarking tool ever. I want to solve existing problems, not artificially created ones. But I hardly ever get some feedback from our users.

[WojciechNagorski]

I pushed PR #1131 for this issue. More details are there.