Benchmarkdotnet: Results table should be a GitHub Flavored Markdown table

[svick]

Currently, the results table printed to the console looks something like this:

    Method |      Mean |    StdDev |    Median |
---------- |---------- |---------- |---------- |
 OldSchool | 2.6605 ms | 0.0601 ms | 2.6675 ms |
    Modern | 2.5060 ms | 0.1050 ms | 2.5105 ms |

This is almost, but not quite, the format used for tables in GitHub Flavored Markdown and if copy-pasted directly into GitHub renders broken:

Method |      Mean |    StdDev |    Median |

---------- |---------- |---------- |---------- |
OldSchool | 2.6605 ms | 0.0601 ms | 2.6675 ms |
Modern | 2.5060 ms | 0.1050 ms | 2.5105 ms |

To make it a GFM table, prepending a pipe to each line seems to work:

|    Method |      Mean |    StdDev |    Median |
|---------- |---------- |---------- |---------- |
| OldSchool | 2.6605 ms | 0.0601 ms | 2.6675 ms |
|    Modern | 2.5060 ms | 0.1050 ms | 2.5105 ms |

Which renders as:

| Method | Mean | StdDev | Median |
|---------- |---------- |---------- |---------- |
| OldSchool | 2.6605 ms | 0.0601 ms | 2.6675 ms |
| Modern | 2.5060 ms | 0.1050 ms | 2.5105 ms |

I think the format of the table should be changed to this, so that it can be easily copy-pasted into GitHub (and any other site that supports the same table format).

[adamsitnik]

@AndreyAkinshin @alinasmirnova sorry for stealing the issue, but it was annoying me since quite a long time ;)

[mattwarren]

Has this issue regressed? I'm seeing quite a few people posting tables that don't have the correct format?

For example see this comment, that is using BenchmarkDotNet=v0.10.14 https://github.com/dotnet/coreclr/issues/17932#issuecomment-388008331 (anther example is here https://github.com/dotnet/coreclr/pull/17928#issuecomment-387810897)

The first link has tables like this (raw view):

  Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |
-------- |---------:|----------:|----------:|-------:|----------:|
 ToArray | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 |  39.13 KB |
 ToArray | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 |  39.13 KB |
 ToArray | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 |  39.13 KB |
 ToArray | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 |  39.13 KB |
 ToArray | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 |  39.13 KB |

Which renders like this (it's okay, but only by chance, see below):

Method | Mean | Error | StdDev | Gen 0 | Allocated |
-------- |---------:|----------:|----------:|-------:|----------:|
ToArray | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 | 39.13 KB |
ToArray | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 | 39.13 KB |
ToArray | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 | 39.13 KB |
ToArray | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 | 39.13 KB |
ToArray | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 | 39.13 KB |

But if the benchmark methods names were longer, like this (raw view):

        Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |
-------------- |---------:|----------:|----------:|-------:|----------:|
 ToArrayLonger | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 |  39.13 KB |
 ToArrayLonger | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 |  39.13 KB |
 ToArrayLonger | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 |  39.13 KB |
 ToArrayLonger | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 |  39.13 KB |
 ToArrayLonger | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 |  39.13 KB |

It would render like this (i.e. borked!!):

    Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |

-------------- |---------:|----------:|----------:|-------:|----------:|
ToArrayLonger | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 | 39.13 KB |
ToArrayLonger | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 | 39.13 KB |
ToArrayLonger | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 | 39.13 KB |
ToArrayLonger | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 | 39.13 KB |
ToArrayLonger | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 | 39.13 KB |

I think the only way to guarantee it always works is to include the | at the beginning of each line?

i.e. (raw view)

|        Method |     Mean |     Error |    StdDev |  Gen 0 | Allocated |
|-------------- |---------:|----------:|----------:|-------:|----------:|
| ToArrayLonger | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 |  39.13 KB |
| ToArrayLonger | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 |  39.13 KB |
| ToArrayLonger | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 |  39.13 KB |
| ToArrayLonger | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 |  39.13 KB |
| ToArrayLonger | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 |  39.13 KB |

which renders as:

| Method | Mean | Error | StdDev | Gen 0 | Allocated |
|-------------- |---------:|----------:|----------:|-------:|----------:|
| ToArrayLonger | 22.57 us | 0.4560 us | 0.1184 us | 9.5215 | 39.13 KB |
| ToArrayLonger | 22.17 us | 1.0280 us | 0.2669 us | 9.5215 | 39.13 KB |
| ToArrayLonger | 22.34 us | 0.8832 us | 0.2294 us | 9.5215 | 39.13 KB |
| ToArrayLonger | 22.28 us | 0.4800 us | 0.1247 us | 9.5215 | 39.13 KB |
| ToArrayLonger | 22.36 us | 0.7602 us | 0.1975 us | 9.5215 | 39.13 KB |

[mattwarren]

Also, I wonder if it's also nice to right-align the 'method' column, i.e. change the header row from:

|-------------- |---------:|----------:|----------:|-------:|----------:|

to (extra ':' in the 1st column)

|--------------:|---------:|----------:|----------:|-------:|----------:|

[svick]

@mattwarren If you look at the commit that fixed this issue, https://github.com/dotnet/BenchmarkDotNet/commit/fe33dd8d724706002a21b6e7e7ee92d403e36502, it fixed MarkdownExporter.GitHub, but I think you have to explicitly opt-in to use that one.

[mattwarren]

@svick thanks, that explains it!

[Mike-E-angelo]

Wow very cool @mattwarren. Thank you for pointing that out. I didn't realize that was markdown, but it's very obvious now. I will check out that MarkdownExporter.GitHub too, @svick. Lots of great things here!

[Mike-E-angelo]

FWIW looks like there is a -report-github.md file that is generated along with a .csv and .html file with every run, using default settings. So it seems you don't even need to opt-in for the goodness here. 👍