Benchmarkdotnet: Introduce BenchmarkDotNet.Annotations

[AndreyAkinshin]

We should move all API which includes attributes and configs to a separate NuGet package BenchmarkDotNet.Annotations.
Benefits:

• This package can target .NETStandard 1.0, so it will be possible to declare benchmarks in .NETStandard projects
• We should be able to run benchmarks from such projects not only managed API (BenchmarkRunner.Run), but also via command line (like dotnet benchmark, see https://github.com/dotnet/BenchmarkDotNet/issues/213)
• It will be possible to implement adapters for other benchmark attribute systems (like xunit-performance)

[AndreyAkinshin]

@adamsitnik, what do you think about this idea?

[adamsitnik]

I really like this idea! It looks like it's a must-have for some scenarios like "I have this piece of IL, please just run it for framework X".

[adamsitnik]

@AndreyAkinshin can I take this issue? it's blocking me

moreover, I have an additional idea: maybe we keep all the types from BenchmarkDotNet.Annotations in a flat BenchmarkDotNet namespace? So when the user puts using BenchmarkDotNet in the code file it's possible to use all our features OOTB without forgetting about the extra Alt+Enter/Ctrl+. to add all the includes? I know it's a breaking change, but I think that our users would like it.

[AndreyAkinshin]

can I take this issue? it's blocking me

Sure.

[AndreyAkinshin]

@adamsitnik, I think, it makes sense to release it in v0.11.x because it contains some breaking changes (with a few other things which also contain breaking changes; I will create corresponding changes soon).

[AndreyAkinshin]

moreover, I have an additional idea: maybe we keep all the types from BenchmarkDotNet.Annotations in a flat BenchmarkDotNet namespace?

Yes, it's a very good idea, I like it!

[Mike-E-angelo]

Is this available on a preview feed somewhere, by chance? Now that I am able to code again (😄) I was looking into creating a .NET Standard library with Benchmark.NET-specific classes. For instance, a strongly-typed wrapper around EnvironmentVariable so that I can keep from having to rely on different text strings for variable names (and values, for that matter).

[adamsitnik]

@Mike-EEE our preview package targets .NET Standard 2.0, you can get if from:

<packageSources>
  <add key="bdn-nightly" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[Mike-E-angelo]

Beautimous. Thank you, @adamsitnik. 👍

[WojciechNagorski]

@adamsitnik Can I help with this issue or is it already done?
After that I could help in #213

[adamsitnik]

It's partially done - I have ported BDN to .NET Standard 2.0.

Before you would start moving the attributes and configs to new library I need to finish #912 first. But I won't have time for it in the next 10 days, so I would say that this issue is currently blocked.

[WojciechNagorski]

Do you have idea what task can I do before?

On Fri, Oct 19, 2018, 15:33 Adam Sitnik [e-mail] wrote:

It's partially done - I have ported BDN to .NET Standard 2.0.

Before you would start moving the attributes and configs to new library I
need to finish #912 https://github.com/dotnet/BenchmarkDotNet/pull/912
first. But I won't have time for it in the next 10 days, so I would say
that this issue is currently blocked.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/BenchmarkDotNet/issues/678#issuecomment-431364549,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQh-j37QlyXeweGP56v3KSgp0BMAJk-pks5umdSrgaJpZM4SlEVd
.

[adamsitnik]

@wojtpl2 if you would like to implement one of the following features it would be awesome

544 - "Diff view for disassembler output" - as of today our disassembler produces a very nice github markdown output https://github.com/dotnet/BenchmarkDotNet/issues/560#issuecomment-430129166 it would be great if you could implement a diff of two assembly listings with an export to GitHub markdown (it has a special "```diff" option) used commonly in CoreCLR by JIT Team members

553 - "New BDN run mode that causes cache misses between iterations" - it would require to add new flag to our Job info and extend the Engine with possibility to clean the cache between iterations

687 - "Implement [Arguments] support for InProcessToolchain" we have two kind of toolchains: the ones that generate and compile C# and the ones that do it with IL. Changing the IL generation is non-trivial, when I implemented [Arguments] support I have updated our C# template file which was easy, but not the code which generates IL.

[WojciechNagorski]

Can you assign me to #544 and #687? I will try to do this next week.

553 it is interesting but I can try to do this later. I think this issue

can take more time.

Of course I would like to do this issue #678 when it's possible. I'm good
at this kind of refactoring.

On Fri, Oct 19, 2018, 16:43 Adam Sitnik [e-mail] wrote:

@wojtpl2 https://github.com/wojtpl2 if you would like to implement one
of the following features it would be awesome

544 https://github.com/dotnet/BenchmarkDotNet/issues/544 - "Diff view

for disassembler output" - as of today our disassembler produces a very
nice github markdown output #560 (comment)
https://github.com/dotnet/BenchmarkDotNet/issues/560#issuecomment-430129166
it would be great if you could implement a diff of two assembly listings
with an export to GitHub markdown (it has a special "```diff" option) used
commonly in CoreCLR by JIT Team members

553 https://github.com/dotnet/BenchmarkDotNet/issues/553 - "New BDN

run mode that causes cache misses between iterations" - it would require to
add new flag to our Job info and extend the Engine with possibility to
clean the cache between iterations

687 https://github.com/dotnet/BenchmarkDotNet/issues/687 - "Implement

[Arguments] support for InProcessToolchain" we have two kind of toolchains:
the ones that generate and compile C# and the ones that do it with IL.
Changing the IL generation is non-trivial, when I implemented [Arguments]
support I have updated our C# template file which was easy, but not the
code which generates IL.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dotnet/BenchmarkDotNet/issues/678#issuecomment-431387012,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AQh-j6vMkEY-5sPojoWvtu832Wl2PcS1ks5umeUHgaJpZM4SlEVd
.

[adamsitnik]

@wojtpl2 it looks like @ig-sinicyn is going to solve #687 with #919

@wojtpl2 we are going to assign you to #544

[adamsitnik]

We have introduced the Annotations quite a while ago, I am closing the issue.