Deno: Feature request: Add 'only' flag to bench API

[cknight]

Deno's testing API has a very useful only option, to only run that one test. It would be great to have this same option in the bench API allowing a single benchmark test to be run. My use case is that I have lots of benchmark tests which can take awhile to run and I wish sometimes to just run one of them to examine performance of just that variant.

[bartlomieju]

I think it's a very sensible future and should be straight-forward to implement. Marking as good first issue.

[jopemachine]

I want to try to resolve the issue.
But It seems that there is already an only option in std/testing/bench.ts, so I'm confused..

Maybe I have missed something because I'm a newbie to deno.
I beg your pardon for my lack of understanding.

May I ask for a little more explanation of where the option should be implemented?

[cknight]

hi @jopemachine , welcome to the deno community. Apologies for my sparse issue. Let me explain in more detail.

From the bench docs:

Filtering can be applied by setting BenchmarkRunOptions.only and/or BenchmarkRunOptions.skip to regular expressions matching benchmark names.

This means coming up with a regular expression to match the benchmarks you want to run. This is not always straightforward and can be awkward matching multiple tests.

Deno's test framework allows the specification of an only flag in the test definition itself (in addition to command line regular expression filtering). It works like this:

test({
  name: "Example use of 'only' flag",
  only: true,
  fn() {
    assertEquals(1, 1);
  },
});

In this scenario, only tests with the only flag will be run.

Ideally we would have this option for bench definitions as well like this:

bench({
  name: "Example use of 'only' flag in bench definition",
  only: true,
  runs: 100,
  func(b: BenchmarkTimer): void {
    b.start();

    for (let i = 0; i < 10000; i++) {
      console.log("Hello world");
    }

    b.stop();
  },
});

Hope this makes sense.

[littletof]

@jopemachine @bartlomieju @cknight how should this feature work together with the skip, and only filter in runBenchmarks?

So if i have benches like:

• itHasOnly
• simpleBench
• extraBench
• extraBenchWithOnly

and run runBenchmarks({ only: /Bench/, skip: /extra/ }) which benchmarks should be ran?
Basically which option is stronger?

[jopemachine]

@jopemachine @bartlomieju how should this feature work together with the skip, and only filter in runBenchmarks?

So if i have benches like:

• itHasOnly
• simpleBench
• extraBench
• extraBenchWithOnly

and run runBenchmarks({ only: /Bench/, skip: /extra/ }) which benchmarks should be ran?
Basically which option is stronger?

In my PR (#8237), if there are benches with only option,
Only these options which have "Bench" will be run.
(In your example, only extraBenchWithOnly will be run)

Otherwise (if there is not bench which have only option, that is, without extraBenchWithOnly), extraBench and simpleBench will be run because they have "Bench" in their bench name.

[jopemachine]

@jopemachine @bartlomieju how should this feature work together with the skip, and only filter in runBenchmarks?

So if i have benches like:

• itHasOnly
• simpleBench
• extraBench
• extraBenchWithOnly

and run runBenchmarks({ only: /Bench/, skip: /extra/ }) which benchmarks should be ran?
Basically which option is stronger?

I'm sorry, I missed skip option.

I think any tests will not be run because of skip option.

and without extraBenchWithOnly, only simpleBench will be run.

---

So, I think only option in bench argument would be most strong at this PR.

[littletof]

Thinking a bit about it, deno test purposefully exits with fail at the end, if only was used.
Shouldn't benching have some similar outcome, or at least indicate it in the output? (in the terminal, and some other way, in case silent is used, possibly in BenchmarkRunResult or with an exception?)

[jopemachine]

Thinking a bit about it, deno test purposefully exits with fail at the end, if only was used.
Shouldn't benching have some similar outcome, or at least indicate it in the output? (in the terminal, and some other way, in case silent is used, possibly in BenchmarkRunResult or with an exception?)

This is the result of the benchmark test code execution.

Check file:///Users/igyubong/Desktop/deno/t.ts
running 1 benchmark ...
benchmark runs100ForIncrementX1e6-2 ... 
    100 runs avg: 0.26ms
benchmark result: DONE. 1 measured; 2 filtered

In the above output, how about changing the benchmark result DONE to FILTERED (with red color) to warn users if there are some filtered benchs?