Benchmarkdotnet: Implement export to xml

[mattwarren]

Note: this should be done without introducing a .dll or Nuget dependency to the core of BenchmarkDotNet. For example the Json export was done with SimpleJson as it is a single .cs file

[Teknikaali]

I can do this.

The previously mentioned Json export seems to be a good example to follow how to implement this.

Some steps I came up with after exploring the repo:

• [x] Figure out an alternative way to serialize Summary-object to XML without XmlSerializer
• [x] Create the new XmlExporter-class by inheriting from ExporterBase
• [x] Include XmlExporter in ApprovalTests
• [x] Define how the XML output should look like

@mattwarren @AndreyAkinshin are there any other things I should take into account before diving in too deep?

Taking the lower level path by using XmlWriter rather than the "now-unavailable-in-netcoreapp1.1" XmlSerializer could be a viable way here.

Currently no XML-alternative exists for SimpleJson that I'd know of, so that leaves me to create a strict XML serialization of the Summary-type. I'll make the serialization part "easy to change", if we'll some day see re-addition of XmlSerializer-class to .NET Core without the need of the extra NuGet dependency.

I start setting up the dev environment!

[adamsitnik]

are there any other things I should take into account before diving in too deep?

@Teknikaali I am currently working on making our .Core package .NET Standard compatible, so if you have to add any new dependency please make sure that it has NuGet package that targets .NET Standard 1.3 or lower

good luck!

[Teknikaali]

@adamsitnik Thanks for the heads-up!

The only NuGet package that (_in theory_) could make the implementation a bit easier, would be System.Xml.XmlSerializer. It should support .NET Standard 1.3 looking at its dependencies list if I'm not all mistaken.

But there's a small catch why the XmlWriter-path seems a bit better than using the XmlSerializer NuGet package:
XmlSerializer would need many changes in the codebase.. just to get the XML serialization to work "in a bit nicer way". In my opinion that is a no-go because of too many changes in other classes just for a single feature.

Here are at least two of the needed changes that would have to be made if we'd like to get the NuGet XmlSerializer to work:

1. Summary and _related classes_ would have to have public set-methods on each property
2. Summary and _related classes_ would have to have default ctor
   (Related classes: Benchmark, BenchmarkReport, HostEnvironmentInfo ...)

I think using XmlSerializer might require too major changes in the codebase, that an alternative way should be used instead? I'm not really sure how things have changed as this issue is dated all the way back to 2016.

One example of "fitting complex class XML serialization into one file like SimpleJson" that I passed by while exploring the topic was this XmlSerializer implementation from 2007. For the heck of it, I took a quick glance at it with ApiPort I saw a result of ~50% compatibility with .NET Core 1.3. With some tweaks I got it to compile and run on 1.3, but the output wasn't really something I'd consider usable..

[Teknikaali]

I've had the dev environment up and running for a while now.

After adding a stub for XmlExporter and including it in ApprovalTests 3 tests started to show red.
What I found out was that ApprovalTest.Exporters.Invariant.approved.txt (also en-US and ru-RU versions) have a couple of missing line breaks like this:

r180: }############################################
r190: JsonExporter-brief-compressed

..when they probably should be like this:

r180: }
r180: ############################################
r181: JsonExporter-brief-compressed

.. also the "full-compressed" has this same thing:

r330: [...]Nanoseconds":1}]}]}############################################

..when it probably should be like this:

r330: [...]Nanoseconds":1}]}]}
r331: ############################################

JsonExporter seems to write to log using logger.Write("...") rather than logger.WriteLine("...").
This can be fixed by changing Write to WriteLine on JsonExporterBase r81.
Should this be made its own issue, or do I just fix it in the same PR I introduce the XmlExporter?
Pinging @mattwarren if using WriteLine to fix this is ok.

[adamsitnik]

@Teknikaali thanks for doing such a great job with the research!

What I found out was that ApprovalTest.Exporters.Invariant.approved.txt (also en-US and ru-RU versions) have a couple of missing line breaks like this:

I think that @alinasmirnova is the best person to answer this question.

Should this be made its own issue, or do I just fix it in the same PR I introduce the XmlExporter?

As for me it can be single PR.

I think using XmlSerializer might require too major changes in the codebase, that an alternative way should be used instead?

You are right. And we want to keep our immutability. I don't have any perfect idea for the alternative solution at the moment. Maybe we could create dto class for Summary/Benchmark with public setters and just map it before the serialization?

And also: are you sure that public setters and parameterless ctors are required even if we do the serialization only (I can imagine that they are used for the deserialization)?

[alinasmirnova]

@Teknikaali, yes, you're right, JsonExporter definitely misses a line separator at the end. I'll make a PR (fixed it already).

[Teknikaali]

@adamsitnik dto and mapping sounds like a good idea.

I just tested a bit more the original XmlSerializer and it's true, that it can't serialize an object with private setters and/or empty constructor. Some of the examples on msdn are using classes with _public fields_. Uh.

@alinasmirnova good job :+1: I'll make sure to pull the changes!

Here is a partial mockup of Summary in xml-form:

<?xml version="1.0" encoding="utf-8"?>
<Summary>
    <Title>MockSummary</Title>
    <HostEnvironmentInfo>
        <BenchmarkDotNet>
            <Caption>BenchmarkDotNet</Caption>
            <Version>0.10.x-mock</Version>
        </BenchmarkDotNet>
        <OsVersion>Microsoft Windows NT 10.0.x.mock</OsVersion>
        <Processor>
            <Name>MockIntel(R) Core(TM) i7-6700HQ CPU 2.60GHz</Name>
            <Count>8</Count>
        </Processor>
        <RuntimeVersion>Clr 4.0.x.mock</RuntimeVersion>
        <Architecture>64mock</Architecture>
        <HasAttachedDebugger>False</HasAttachedDebugger>
        <HasRyuJit>True</HasRyuJit>
        <Configuration>CONFIGURATION</Configuration>
        <JitModules>clrjit-v4.6.x.mock</JitModules>
        <Chronometer>
            <Frequency>
                <Hertz>2531248</Hertz>
            </Frequency>
            <Resolution>
                <Nanoseconds>395.062</Nanoseconds>
            </Resolution>
        </Chronometer>
    </HostEnvironmentInfo>
</Summary>

It still needs the actual Benchmarks-part figured out, but I'll just throw this here for some possible input and improvement suggestions. I'll post a more comprehensive listing later.

For this mock I merged some properties (like ProcessorName and ProcessorCount) under a single element (Processor) with the actual properties (Name and Count) as sub elements.

I've left out using _attributes_ altogether, but for example Chronometer could be alternatively expressed like this:

<Chronometer>
    <Frequency unit="Hertz">2531248</Frequency>
    <Resolution unit="Nanoseconds">395.062</Resolution >
<Chronometer>

I would think it is fine to use an attributes in a place like this if needed as "unit" could be interpreted as "data about data".

When comparing MarkdownExporter and JsonExporter-full outputs, it seems like the MarkdownExporter has simpler output and less data in it. Should the xml exporter give out only the most important bits of information like MarkdownExporter does, or should it blow out a full report like the JsonExporter-full?

[adamsitnik]

For this mock I merged some properties

it looks OK to me. @AndreyAkinshin what are your thoughts?

Should the xml exporter give out only the most important bits of information like MarkdownExporter does, or should it blow out a full report like the JsonExporter-full?

Would it be possible to make it similar to JsonExporter which exposes few options?

Full
BriefCompressed
FullCompressed 

[mattwarren]

Would it be possible to make it similar to JsonExporter which exposes few options?

I agree, it would be nice to have the exporters use the same data (although in different formats), so if you use Full with XML and Json you'll get identical fields/value.

[AndreyAkinshin]

@Teknikaali

For this mock I merged some properties (like ProcessorName and ProcessorCount) under a single element (Processor) with the actual properties (Name and Count) as sub elements.

I think that we should keep consistency between json and xml formats. I like the idea of this merge, but you should also fix the JsonExporterBase.ExportToLog method (by the way, parts like "IsValueCreated":true looks like a bug for me; we should call Value for all Lazy<string> properties).

[Teknikaali]

Would it be possible to make it similar to JsonExporter which exposes few options?

I'll make sure to include the same options as the JsonExporter has.

Here is a complete XML output for the Full-style, it should be identical to the Json one if I didn't miss anything:

XmlExporter-full

<?xml version="1.0" encoding="utf-8"?>
<Summary>
  <Title>MockSummary</Title>
  <HostEnvironmentInfo>
    <BenchmarkDotNetCaption>BenchmarkDotNet</BenchmarkDotNetCaption>
    <BenchmarkDotNetVersion>0.10.x-mock</BenchmarkDotNetVersion>
    <OsVersion>Microsoft Windows NT 10.0.x.mock</OsVersion>
    <ProcessorName>MockIntel(R) Core(TM) i7-6700HQ CPU 2.60GHz</ProcessorName>
    <ProcessorCount>8</ProcessorCount>
    <RuntimeVersion>Clr 4.0.x.mock</RuntimeVersion>
    <Architecture>64mock</Architecture>
    <HasAttachedDebugger>False</HasAttachedDebugger>
    <HasRyuJit>True</HasRyuJit>
    <Configuration>CONFIGURATION</Configuration>
    <JitModules>clrjit-v4.6.x.mock</JitModules>
    <DotNetCliVersion>1.0.x.mock</DotNetCliVersion>
    <ChronometerFrequency>
      <Hertz>2531248</Hertz>
    </ChronometerFrequency>
    <HardwareTimerKind>Tsc</HardwareTimerKind>
  </HostEnvironmentInfo>
  <Benchmarks>
    <Benchmark>
      <DisplayInfo>MockBenchmarkClass.Foo: LongRun(LaunchCount=3, TargetCount=100, WarmupCount=15)</DisplayInfo>
      <Namespace>BenchmarkDotNet.Tests.Mocks</Namespace>
      <Type>MockBenchmarkClass</Type>
      <Method>Foo</Method>
      <MethodTitle>Foo</MethodTitle>
      <Statistics>
        <N>1</N>
        <Min>1</Min>
        <LowerFence>1</LowerFence>
        <Q1>1</Q1>
        <Median>1</Median>
        <Mean>1</Mean>
        <Q3>1</Q3>
        <UpperFence>1</UpperFence>
        <Max>1</Max>
        <InterquartileRange>0</InterquartileRange>
        <StandardError>0</StandardError>
        <Variance>0</Variance>
        <StandardDeviation>0</StandardDeviation>
        <Skewness>NaN</Skewness>
        <Kurtosis>NaN</Kurtosis>
        <ConfidenceInterval>
          <N>1</N>
          <Mean>1</Mean>
          <StandardError>0</StandardError>
          <Level>L999</Level>
          <Margin>NaN</Margin>
          <Lower>NaN</Lower>
          <Upper>NaN</Upper>
        </ConfidenceInterval>
        <Percentiles>
          <P0>1</P0>
          <P25>1</P25>
          <P50>1</P50>
          <P67>1</P67>
          <P80>1</P80>
          <P85>1</P85>
          <P90>1</P90>
          <P95>1</P95>
          <P100>1</P100>
        </Percentiles>
      </Statistics>
      <Measurements>
        <Measurement>
          <IterationMode>Result</IterationMode>
          <LaunchIndex>1</LaunchIndex>
          <IterationIndex>1</IterationIndex>
          <Operations>1</Operations>
          <Nanoseconds>1</Nanoseconds>
        </Measurement>
      </Measurements>
    </Benchmark>
    <Benchmark>
      <DisplayInfo>MockBenchmarkClass.Bar: LongRun(LaunchCount=3, TargetCount=100, WarmupCount=15)</DisplayInfo>
      <Namespace>BenchmarkDotNet.Tests.Mocks</Namespace>
      <Type>MockBenchmarkClass</Type>
      <Method>Bar</Method>
      <MethodTitle>Bar</MethodTitle>
      <Statistics>
        <N>1</N>
        <Min>1</Min>
        <LowerFence>1</LowerFence>
        <Q1>1</Q1>
        <Median>1</Median>
        <Mean>1</Mean>
        <Q3>1</Q3>
        <UpperFence>1</UpperFence>
        <Max>1</Max>
        <InterquartileRange>0</InterquartileRange>
        <StandardError>0</StandardError>
        <Variance>0</Variance>
        <StandardDeviation>0</StandardDeviation>
        <Skewness>NaN</Skewness>
        <Kurtosis>NaN</Kurtosis>
        <ConfidenceInterval>
          <N>1</N>
          <Mean>1</Mean>
          <StandardError>0</StandardError>
          <Level>L999</Level>
          <Margin>NaN</Margin>
          <Lower>NaN</Lower>
          <Upper>NaN</Upper>
        </ConfidenceInterval>
        <Percentiles>
          <P0>1</P0>
          <P25>1</P25>
          <P50>1</P50>
          <P67>1</P67>
          <P80>1</P80>
          <P85>1</P85>
          <P90>1</P90>
          <P95>1</P95>
          <P100>1</P100>
        </Percentiles>
      </Statistics>
      <Measurements>
        <Measurement>
          <IterationMode>Result</IterationMode>
          <LaunchIndex>1</LaunchIndex>
          <IterationIndex>1</IterationIndex>
          <Operations>1</Operations>
          <Nanoseconds>1</Nanoseconds>
        </Measurement>
      </Measurements>
    </Benchmark>
  </Benchmarks>
</Summary>

[adamsitnik]

looks great!

[adamsitnik]

big thanks to @Teknikaali for implementing the feature!