Benchmarkdotnet: [Suggestion] Specify Setup per benchmark

[BlueRaja]

It'd be nice to be able to do something like this:

public void MySetup() => /* ... */;

[Benchmark(Setup: MySetup)]
public void MyBenchmark() => /* ... */;

My use case: I'd like to test different methods on my custom data structure with different initial setups. I of course don't want the initial setup to get included as part of the benchmark, so right now my only option is to write a different Benchmark class for each setup. This makes the benchmark code ugly, and even worse, forces me (as far as I can tell) to split the results across multiple output files.

[AndreyAkinshin]

It's a good suggestion, thanks! However, I have another idea for the API. Since we have [GlobalSetup], [GlobalCleanup], [IterationSetup], [IterationCleanup] (probably, we will add some additional attributes later). So, I don't want to add all of these attributes as parameters in the [Benchmark] attribute. But we can do something like this:

```cs
[GlobalSetup(Target: nameof(MyBenchmark))]
public void MySetup() => /* ... */;

[Benchmark]
public void MyBenchmark() => /* ... */;
``` So, ifTarget` (probably we have to find a better name, there are too many "Targets" in the source code) is null, attribute will be appied to the all benchmark methods. If the value is specified, it will be applied to only single method.

Optional feature:
```cs
[GlobalSetup(Targets: nameof(MyBenchmark1) + "," + nameof(MyBenchmark2))]
public void MySetup() => /* ... */;

[Benchmark]
public void MyBenchmark1() => /* ... */;

[Benchmark]
public void MyBenchmark2() => /* ... */;
````

@BlueRaja, @adamsitnik, what do you think?

[ig-sinicyn]

@AndreyAkinshin
I'd propose alternate option: pass an object with information about current benchmark as an arg to setup / cleanup methods (if the method has one). This way setup logic will be able to determine current method and job settings and etc.

[BlueRaja]

I prefer my suggestion. You may want several benchmarks to share a single initialization method, but it makes no sense for a single test to have multiple initialization methods.

[adamsitnik]

hmm, I am not sure.

I like @BlueRaja suggestion, but when we want to handle all scenarios: global/iteration setup and cleanup, then our [Benchmark] has four new arguments. I am afraid that it can confuse our users. It has 2 arguments today (Baseline and OperationsPerInvoke). It used to have more, but @AndreyAkinshin has refactored it a long time ago.

In @AndreyAkinshin idea I don't like the string concatenation, but I know that it could be replaced with an array of strings.

@ig-sinicyn I think that passing an object to setup method is not a good idea. Users would need to play with reflection to choose the right methods. I don't think that it would work for most of them.

I have some other ideas:
Add possibility to specify a global/iteration setup/cleanup for given benchmark via attribute. Sth like this:

public void MySetup() => /* ... */;

[Benchmark]
[BenchmarkGlobalSetup(MethodName: nameof(MySetup))]
public void MyBenchmark() => /* ... */;

or maybe we should just allow to compare (put in a single summary) results of benchmarks that are splitted into few types? Sth like this:

class ArrayBenchmarks { }
class SpanBenchmarks { }

BenchmarkRunner.Run(new Set(typeof(ArrayBenchmarks), typeof(SpanBenchmarks));

I am not sure.

[BlueRaja]

To be clear, what I was suggesting was a way to setup per invocation, not per iteration/benchmark/suite.

Eg. to test the speed of Dequeue() for my Priority Queue, I first need something _in_ the queue. Given that I just learned that "iteration" and "invocation" are different, there doesn't seem to currently be a way to do that, without also testing the speed of Clear() and Enqueue() in the same test.

[Edit] In fact, as far as I can tell it will _always_ be true that benchmarks that edit state will have the state-changes carry-over between tests. Meaning currently it's impossible to benchmark any code that changes state without also benchmarking the code to reverse that state.

Surely I can't be the first person to test stateful code? How is everyone else solving this issue?

[BlueRaja]

Oh I see, I misunderstood @AndreyAkinshin's comment here. So IterationSetup is meant to solve this problem, but it only works if you set RunStrategy = Monitoring

That's an acceptable workaround, but I'm going to keep this open because it's silly to have a feature that ostensibly does one thing but is really intended to do another. Don't have a useless "per-iteration setup" that is intended to be used as a "per-invocation setup", just make a per-invocation setup.

[ipjohnson]

I'm definitely interested in a feature like this. As a work around is there a way to get access to the name of the benchmark method being executed in the setup method?

[AndreyAkinshin]

@ipjohnson, sorry there is no a work around for now. I will implement this feature as soon as I find some free time (my schedule is very busy lately). By the way, pull requests are welcome. =)

[AndreyAkinshin]

Fixed in #501