Catch2: Benchmark improvements and feature requests

[horenmar]

Description
This is an overarching issue for Benchmarking-related feature requests. It should serve as a central place to discuss possible extensions of the current benchmarking support and figuring what the users want.

Additional context
Currently opened issues/requests:

• Let user set number of collected samples and in-sample iteration
• Let user divide per-iteration timing -- Here the idea is that each iteration might be working on an N element batch, and thus the reported iteration times should be divided further.
• Let user control the measured time -- This could also be solved by letting the user ask for more iterations/telling user of more iterations.

[RT222]

Good idea. I'd like to add another suggestion : It would be nice to allow the user to customize the benchmarks ouput style. I find the current formatting of the results not very easy to read, and liked the table style of the previous version better.

[jimpark]

Can we make running benchmarks optional and only enabled via cli option? The problem is that running tests now also run benchmarks which take a lot longer to run. I'd like the default behavior of CTest (CMake tests) to run without benchmarking to save build time since unit tests are run with the build.

It would also be nice to have a simple timing feature like the old BENCHMARK. Instead of desiring any real benchmarking, we just want to be able to check that our recent check-in didn't cause some catastrophic degradation in performance.

[poconbhui]

There was some discussion on the Discord server, but I'd like to bring it up here, about the pros and cons of allowing users to set their own elapsed timings, as in #1701 .

I'll start things off, how I see them.

Pros

• Allows user to arbitrarily alter timing
• No need to add extra abstractions as use cases arise
  
  
  
  • i.e. allows users to implement the motivation behind both #1690 (per-iteration timing for benchmarked loops) and #1701 (consistent number of calls to benchmark body in distributed and load-imbalanced code)
  
  
• May encourage messy / unintuitive code

Cons

• Allows user to arbitrarily alter timing
• Requires exposing some internals of the timing (i.e. units of time measurable by the clock), potentially effecting maintainability + portability.
• Requires user to understand something of the benchmark timing and use sane timing modifications.
• Requires some extra thought to ensure the measurement phase of the benchmark works as expected when users can modify elapsed times.

I have plenty of biased comments to add, but I'll let someone else comment first!

[gian21391]

My suggestion is to give the possibility to include custom columns in the results. Something like custom counters in Google Benchmark.

[RT222]

I have two more suggestions :

• Replace --benchmark-no-analysis with --benchmark-analysis, thus disabling analysis by default. I think it would be a better default option because, while being a very cool feature to have, benchmark analysis is slower, harder to read and probably not very useful in a majority of cases.

• Add baselines. It's a very convenient feature to have in a benchmark toolset in order to compare your code performance to a reference.

[VioletGiraffe]

I have not necessarily a request, but a question at first: how do I benchmark code with side effects where each iteration needs to repeat - but not measure - the preparation step?
Say I want to measure std::vector::clear():

BENCHMARK_ADVANCED("clear")(Catch::Benchmark::Chronometer meter) {
            std::vector<int> v(100000, 0);
            meter.measure([&] { v.clear(); return v.size(); });
        };

This is wrong because measure will be called multiple times, and all but the very first call are no-ops.

[Dudly01]

@horenmar what @VioletGiraffe says is something I came across too as a problem as I want to benchmark a function that modifies its input argument. I tried to solve it and this is what I could think of:

• I thought about inserting the _preparation step_ into the measurement and measuring it separately to just substract the result. This feels troublesome if the _preparation step_ consits of setting up more than one variables (or copying those from before the meter.measure([&] { ... }); part.
• Google benchmark has a way to overcome this. I just do not like using it instead of Catch as I feel like it is cluttering my project.
• Maybe being able to set the runs per samples could also overcome this. (At least for me, as my code is only running once per sample during measurement).

Any insight to this problem?