Catch2: Benchmark manual samples and iterations count

[RT222]

Currently, Catch2 Benchmark automatically compute the required iterations count for a benchmark to be accurate, and the sample count seems to be fixed to 100.

This is great. But in some cases, it would be very useful to be able to set this manually, especially the sample count. This is possible in Celero, and it would be great to have it too in Catch2.

[sfranzen]

It looks like the the sample count is already adjustable through the --benchmark-samples option, at least since Catch 2.9.0 I suppose. However, you're right that there is no such option for the iterations per sample, that could be added still.

[RT222]

I didn't know that, thanks.

[VioletGiraffe]

There should be a way to adjust the sample / iteration count automatically based on the estinated runtime. When I'm benchmarking insertion of 1000 items in my container I do want a large number of sample. When I'm inserting 10 million items and it takes 10 seconds for each sample, I don't want 100 samples.

[ruben-arts]

Is it possible to define the benchmark samples in an other form then the --benchmark-samples? I would like to be able to define the samples per benchmark. But I don't know how.