Benchmarkdotnet: Possibility to turn off GC.Collect after each benchmark run

[adamsitnik]

By default MethodInvoker forces full garbage collection after each benchmark invocation.

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

This is of course the desired behaviour. But sometimes users would like to avoid that, for example when the benchmarked methods can benefit from auto-tuning nature of GC.

How to use:

Job.Default.With(
    new GcMode
    {
        Force = false
    });

[adamsitnik]

@AndreyAkinshin could you please do detailed code review? I have introduced changes to our Core (MethodIvoker)

@mattwarren Perhaps this could be nice feature when using MemoryDiagnoser?

[AndreyAkinshin]

for example when the benchmarked methods can benefit from auto-tuning nature of GC.

I think, it is a bad idea. Your benchmark method should produce stable measurements. I think, it can be ok to disable GC.Collects for warmup. However, GC.Collects shouldn't spoil your results for target iterations. In the perfect benchmark, you should do your GC auto-tuning magic in the Setup method. Theoretically, we can do it during warmup, but we should be carefully: it is really easy to do stupid benchmark mistakes.

[mattwarren]

As an interesting counter-argument, see this tweet from Aleksey Shipilëv (JMH Guru), the replies are also worth reading.

[redknightlois]

I do feel the same as Matt on this one. One will pay for the sins of the father in production code, as long as the parent is always the same an extra allocation can throw you over the edge (as it would happen in production). The issue with the GC is that we are not testing spherical chickens in vacuum. So not allow it to run you will never know what is the real time to expect. Maybe I am biased but my usual benchmarking is on algorithms that run in the nanosecond level with microoptimizations of the kind to avoid pipeline stalls.

[adamsitnik]

First of all: I don't want to change the default behawior, but only allow "super" users to do their job right. For example somebody that wants to compare allocators, measure finalization side-effects etc.

We all know how GC in .NET works: when CLR tries to allocate new small object and there is no memory available in Gen 0, the Gen 0 is collected. Everything that survives goes to Gen 1. And so on, bla bla bla.

The problem that I see with forcing GC after each benchmark run is when our Benchmark allocated less memory than 100% of Gen 0 size for single run. What happens when just after that we force GC.Collect? It clears whole Gen 0. Next benchmark run starts with empty Gen 0. But in the real world scenario, next invocation would start with non-empty Gen 0 and more probably invoke Gen 0 collection.

Second scenario: the benchmark is allocating objects with Finalizers, for example Tasks. What would happen with these objects if there was no GC.Collect => GC.WaitForPendingFinalizers => GC.Collect? They would survive at least one collection. If we force, we are not able to see it at all.

I did some tests, and what I see from the results that the difference for the median time is usually small but the difference for GC stats (for example when using MemoryDiagnoser) can be quite big.

Let's consider:

private object willNotRequireClosure = new object();

[Benchmark(Baseline = true)]
public Task ContinueWith()
{
    return Task.CompletedTask
        .ContinueWith(
            (_, state) =>
            {
                GC.KeepAlive(state);
            }, willNotRequireClosure);
}

Results (Forcing excludes Finalization Survivors (0 Gen 1 collections))

              Method |                GarbageCollection |      Median |      StdDev | Scaled |  Gen 0 | Gen 1 | Gen 2 | Bytes Allocated/Op |
-------------------- |--------------------------------- |------------ |------------ |------- |------- |------ |------ |------------------- |
        ContinueWith |      DontForce Concurrent Server | 709.3613 ns |  78.0581 ns |   1.00 |   4.00 |  2.00 |     - |              53,19 |
        ContinueWith |                Concurrent Server | 806.5756 ns |  90.0179 ns |   1.00 |   3.09 |     - |     - |              58,96 |

[windhandel]

I've seen production Gen 0 GCs that take 4+ seconds.

Where did this land and is there a way to change this default behavior via configuration?

The other seemingly odd behavior is that the Gc is defaulted to run in Workstation, except production is always going to be Server. Why would the default be set to a non-production mode if our goal is to measure production type behavior?

[windhandel]

Additionally, the GcConcurrent value seems to default to concurrent behavior, despite everything I've seen from production Azure environments does not seem to run with this behavior.

[abelbraaksma]

@adamsitnik This is certainly interesting, age valuable in certain scenarios, what attributes have been added to make this possible, or was it ultimately scrapped?

Was this the final way of doing this?

c# Job.Default.With( new GcMode { Force = false });

[adamsitnik]

Gc is defaulted to run in Workstation, except production is always going to be Server.

The defaults depend on the application type. GC Server mode is not a silver bullet and since BenchmarkDotNet runs every benchmark in a dedicated console app, we don't enforce any custom GC settings and use the defaults for the console app.

Where did this land and is there a way to change this default behavior via configuration?

There is a way to configure all known GC settings. To find out how you either need to visit https://benchmarkdotnet.org/ and type GC in the search bar or use IntelliSense in your IDE.

[abelbraaksma]

To find out how you either need to visit https://benchmarkdotnet.org/ and type GC i

Haha, you're absolutely right! I was just browsing the github code, but should've checked that before asking, tx!

https://benchmarkdotnet.org/api/BenchmarkDotNet.Attributes.GcForceAttribute.html#BenchmarkDotNet_Attributes_GcForceAttribute