Crystal: Improve reporting format of Benchmark.ips

Created on 4 Jan 2018  Â·  5Comments  Â·  Source: crystal-lang/crystal

With #5522 merged, the reports from Benchmark.ips look like this:

  one   1.33M (753.32ns) (± 7.90%)  624 B/op  104.55× slower
  two   3.62M (276.15ns) (± 2.50%)  209 B/op   38.33× slower
three 138.79M (  7.21ns) (± 9.09%)    0 B/op         fastest

I've always thought the output format is a bit misleading, as values like 1.33 M can easily be interpreted as a size in bytes - this is for example used in the output of ls -lh.

Now that there is really a memory size included in the IPS output, this feels even more pressing.
I have also a few other ideas, please feel free to add suggestions.

  • Add a unit op/s for the first field, making it look like this:
  one   1.33M op/s (753.32ns) (± 7.90%)  624 B/op  104.55× slower
  two   3.62M op/s (276.15ns) (± 2.50%)  209 B/op   38.33× slower
three 138.79M op/s (  7.21ns) (± 9.09%)    0 B/op         fastest
  • The two parenthesised values for time per op and stdev look strange. Maybe we should just remove ) ( in between. It wouldn't change the
  one   1.33M (753.32ns ± 7.90%)  624 B/op  104.55× slower
  two   3.62M (276.15ns ± 2.50%)  209 B/op   38.33× slower
three 138.79M (  7.21ns ± 9.09%)    0 B/op         fastest
  • Larger values for allocated bytes are hard to read. It should probably better be formatted with an SI prefix as it is already used for the number of instructions per second. It would probably be worth extracting this number formatting to a more general method available in the stdlib.
  • Long names make the layout look really werid. There could be a linebreak inserted after a name exceeding X number of characters. But this is more effort and probably not really worth it.
feature stdlib

Most helpful comment

Looks good!

I'd like the time per op to be more explicit, so ns/ops, something like:

  one   1.33M op/s (753.32 ns/op ± 7.90%)  624 B/op  104.55× slower
  two   3.62M op/s (276.15 ns/op ± 2.50%)  209 B/op   38.33× slower
three 138.79M op/s (  7.21 ns/op ± 9.09%)    0 B/op         fastest

Maybe even be more explicit with B/op and change it to bytes/op (maybe B is a bit cryptic):

  one   1.33M op/s (753.32 ns/op ± 7.90%)  624 bytes/op  104.55× slower
  two   3.62M op/s (276.15 ns/op ± 2.50%)  209 bytes/op   38.33× slower
three 138.79M op/s (  7.21 ns/op ± 9.09%)    0 bytes/op         fastest

It still fits on a screen, 72 chars in the above example.

All 5 comments

Looks good!

I'd like the time per op to be more explicit, so ns/ops, something like:

  one   1.33M op/s (753.32 ns/op ± 7.90%)  624 B/op  104.55× slower
  two   3.62M op/s (276.15 ns/op ± 2.50%)  209 B/op   38.33× slower
three 138.79M op/s (  7.21 ns/op ± 9.09%)    0 B/op         fastest

Maybe even be more explicit with B/op and change it to bytes/op (maybe B is a bit cryptic):

  one   1.33M op/s (753.32 ns/op ± 7.90%)  624 bytes/op  104.55× slower
  two   3.62M op/s (276.15 ns/op ± 2.50%)  209 bytes/op   38.33× slower
three 138.79M op/s (  7.21 ns/op ± 9.09%)    0 bytes/op         fastest

It still fits on a screen, 72 chars in the above example.

I definitely prefer the last snippet @asterite :+1: It's cleaner and as descriptive as possible without being cryptic :cake:

I actually find the current output far far easier to skim with my eyes, and less noisy than the new proposed output. Whether that's worth the confusion to people who are new to Benchmark.ips, i'm not sure.

@RX14 That's probably a bit of being used to the current output - at least to some point.

I'm not sure if we need explict ns/op - it should be pretty obvious what ns refers to. B vs bytes I'm not really decided... I'd probably slighlty prefer the shorter B.

Weird idea... what about adding a header?

Name   ops/s    time/op           memory/op  ratio
-----------------------------------------------------------
  one    1.33M  753.32ns ± 7.90%  624 B      104.55× slower
  two    3.62M  276.15ns ± 2.50%  209 B       38.33× slower
three  138.79M    7.21ns ± 9.09%    0 B             fastest

But I'm fine with any suggestion here, they all seem to be an improvement.

Was this page helpful?
0 / 5 - 0 ratings