Benchmarkdotnet: Simple API for path customization of BenchmarkDotNet.Artifact and job folders

[AndreyAkinshin]

Sometimes we have troubles with UnauthorizedAccessException (we don't have access to the current directories) or long paths. I think, it would be great to provide a nice API which allows to customzie BenchmarkDotNet working directory (which will contain all the generated files and folders).

[adamsitnik]

which will contain all the generated files and folders

@AndreyAkinshin do you mean only the results (.csc, .html and etc) and logs (.log)? then such change is easy

but if you also mean (.exe, .notcs and .config) then not so easy because today we use benchmark.Target.Type.GetTypeInfo().Assembly.Location where something (like msbuild) has produced the exe which makes assembly loading very easy for us (everything is in the current directory). If we want to produce the .exe somewhere else then most probably we would need to copy some files there or play with the CurrentDirectory of the benchmark process

[AndreyAkinshin]

Hmm. In the perfect world, I want to have an ability to specify a folder for all generated files. But we can start with the artifacts folder.

[daveaglick]

I hit this trying to set up some benchmarks in LINQPad. Specifically, it's trying to write to "C:Program Files (x86)LINQPad5BenchmarkDotNet.Artifacts" which is locked down since it's in the "Program Files" folder. I can work around it, but a simple API to direct file writes elsewhere would be helpful.

[jhorv-scs]

I hit upon this problem in LINQPad. In lieu of a BenchmarkDotNet API for configuring, I found that setting System.Environment.CurrentDirectory to another existing directory will change the benchmark output location. Below is some general code for setting this to the same location and name as your LINQPad query:

void Main()
{
    Environment.CurrentDirectory = Directory.CreateDirectory(GetBenchmarkOutputPath()).FullName;
    // ...
}

static string GetBenchmarkOutputPath()
{
    var currentQueryDirectory = Path.GetDirectoryName(LINQPad.Util.CurrentQueryPath);
    var currentQueryName = Path.GetFileNameWithoutExtension(LINQPad.Util.CurrentQueryPath);
    var result = Path.Combine(currentQueryDirectory, currentQueryName);
    result += "_" + DateTime.Now.ToString("yyyyMMdd-HHmmss");
    return result;
}

[seesharper]

I am hitting something similar trying to execute a benchmark from a C# script where the underlying generated assembly for the script has no location.

System.AggregateException: One or more errors occurred. (One or more errors occurred. (Object reference not set to an instance of an object.)) (One or more errors occurred. (Object reference not set to an instance of an object.)) (O
bject reference not set to an instance of an object.) (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object.                                 
   at BenchmarkDotNet.Toolchains.DotNetCli.DotNetCliGenerator.GetArtifactsToCleanup(Benchmark benchmark, ArtifactsPaths artifactsPaths)                                                                                                 
   at BenchmarkDotNet.Toolchains.GeneratorBase.GenerateProject(Benchmark benchmark, ILogger logger, String rootArtifactsFolderPath, IConfig config, IResolver resolver)                                                                 
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(Benchmark benchmark, ReadOnlyConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)                                                       
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.<>c__DisplayClass7_0.<BuildInParallel>b__2(Benchmark benchmark)                                                                                                                       
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)                                                                                                    
   at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()                                                                                                                                                                       
   at System.Linq.Parallel.SpoolingTaskBase.Work()                                                                                                                                                                                      
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)                                                                                                                                                                            
   at System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o)                                                                                                                                                                      
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)                                                                                                                  
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)                                                                                                                                                         
   --- End of inner exception stack trace ---                                                                                                                                                                                           
   at System.Linq.Parallel.QueryTaskGroupState.QueryEnd(Boolean userInitiatedDispose)                                                                                                                                                   
   at System.Linq.Parallel.SpoolingTask.SpoolStopAndGo[TInputOutput,TIgnoreKey](QueryTaskGroupState groupState, PartitionedStream`2 partitions, SynchronousChannel`1[] channels, TaskScheduler taskScheduler)                           
   at System.Linq.Parallel.DefaultMergeHelper`2.System.Linq.Parallel.IMergeHelper<TInputOutput>.Execute()                                                                                                                               
   at System.Linq.Parallel.MergeExecutor`1.Execute[TKey](PartitionedStream`2 partitions, Boolean ignoreOutput, ParallelMergeOptions options, TaskScheduler taskScheduler, Boolean isOrdered, CancellationState cancellationState, Int32 
queryId)                                                                                                                                                                                                                                
   at System.Linq.Parallel.PartitionedStreamMerger`1.Receive[TKey](PartitionedStream`2 partitionedStream)                                                                                                                               
   at System.Linq.Parallel.UnaryQueryOperator`2.UnaryQueryOperatorResults.GivePartitionedStream(IPartitionedStreamRecipient`1 recipient)                                                                                                
   at System.Linq.Parallel.QueryOperator`1.GetOpenedEnumerator(Nullable`1 mergeOptions, Boolean suppressOrder, Boolean forEffect, QuerySettings querySettings)                                                                          
   at System.Linq.Parallel.QueryOpeningEnumerator`1.OpenQuery()                                                                                                                                                                         
   at System.Linq.Parallel.QueryOpeningEnumerator`1.MoveNext()                                                                                                                                                                          
   at System.Linq.ParallelEnumerable.ToDictionary[TSource,TKey,TElement](ParallelQuery`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)                                                              
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.BuildInParallel(ILogger logger, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver, Benchmark[] benchmarks, ReadOnlyConfig config, StartedClock& globalChron
ometer)                                                                                                                                                                                                                                 
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(BenchmarkRunInfo benchmarkRunInfo, ILogger logger, String title, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver, List`1 artifactsToCleanup)         
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(BenchmarkRunInfo benchmarkRunInfo, Func`2 toolchainProvider)                                                                                                                      
   at Submission#0.<<Initialize>>d__0.MoveNext() in C:\Github\fresh\dotnet-script\src\Dotnet.Script.Tests\TestFixtures\Issue221\Issue221.csx:line 6                                                                                     
--- End of stack trace from previous location where exception was thrown ---                                                                                                                                                            
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()                                                                                                                                                                    
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)                                                                                                                                    
   at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.<RunSubmissionsAsync>d__9`1.MoveNext()                                                                                                                                      
---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.                                                                                                                          
   at BenchmarkDotNet.Toolchains.DotNetCli.DotNetCliGenerator.GetArtifactsToCleanup(Benchmark benchmark, ArtifactsPaths artifactsPaths)                                                                                                 
   at BenchmarkDotNet.Toolchains.GeneratorBase.GenerateProject(Benchmark benchmark, ILogger logger, String rootArtifactsFolderPath, IConfig config, IResolver resolver)                                                                 
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(Benchmark benchmark, ReadOnlyConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)                                                       
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.<>c__DisplayClass7_0.<BuildInParallel>b__2(Benchmark benchmark)                                                                                                                       
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)                                                                                                    
   at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()                                                                                                                                                                       
   at System.Linq.Parallel.SpoolingTaskBase.Work()                                                                                                                                                                                      
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)                                                                                                                                                                            
   at System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o)                                                                                                                                                                      
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)                                                                                                                  
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)<---                                                                                                                                                     

---> (Inner Exception #1) System.NullReferenceException: Object reference not set to an instance of an object.                                                                                                                          
   at BenchmarkDotNet.Toolchains.DotNetCli.DotNetCliGenerator.GetArtifactsToCleanup(Benchmark benchmark, ArtifactsPaths artifactsPaths)                                                                                                 
   at BenchmarkDotNet.Toolchains.GeneratorBase.GenerateProject(Benchmark benchmark, ILogger logger, String rootArtifactsFolderPath, IConfig config, IResolver resolver)                                                                 
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(Benchmark benchmark, ReadOnlyConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)                                                       
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.<>c__DisplayClass7_0.<BuildInParallel>b__2(Benchmark benchmark)                                                                                                                       
   at System.Linq.Parallel.PartitionedDataSource`1.ListContiguousIndexRangeEnumerator.MoveNext(T& currentElement, Int32& currentKey)                                                                                                    
   at System.Linq.Parallel.StopAndGoSpoolingTask`2.SpoolingWork()                                                                                                                                                                       
   at System.Linq.Parallel.SpoolingTaskBase.Work()                                                                                                                                                                                      
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)                                                                                                                                                                            
   at System.Linq.Parallel.QueryTask.<>c.<.cctor>b__10_0(Object o)                                                                                                                                                                      
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)                                                                                                                  
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)<---                                                                                                                                                     

[ViktorHofer]

I would love to be able to customize the BenchmarkDotNet.Artifacts path not just by an API but as a CLI option. Something like:

dotnet run -c Release benchmark1.dll artifacts=./master-regex-list

That would allow me to run a benchmark multiple times without overriding the results.

[adamsitnik]

@ViktorHofer Done! As soon as this build is over, you should be able to download 0.10.12.430 version with this new feature from our CI feed.

<packageSources>
  <add key="bdn-nightly" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[ArtifactsPath(@"C:\here")]
[DryJob]
public class Simple
{
    [Benchmark]public void Nothig() { }
}

or as suggested via the command line: dotnet run -c Release -f net46 -- --artifacts="C:\AdamTest" --class=Cpu_Atomics

or via the ConfigExtensions: config.WithArtifactsPath(thePath)

[ViktorHofer]

Adam you are fast!! Thanks a lot! Happy to have you on our team 👍

[adamsitnik]

@ViktorHofer thanks! I will contact you soon about local Core CLR/FX builds

[filipw]

@adamsitnik is there a way to run against in-memory assemblies though?

[adamsitnik]

@filipw hi Filip! You should try our InProcessToolchain which does not compile and spawn new processes but instead generates new types using IL Emit. Here is a good description of it

[isabr85]

Hi @adamsitnik ,
I am using BenchmarkDotNet in Azure Function. When I am triggering this function I am getting an error that tells me "Access to the path '........BenchmarkDotNet.Artifactsresults' is denied"
Can I disable the writing to the directory?

[adamsitnik]

Hi @isabr85

To disable all exporters you need to define a config with no exporters. This should work:

var noExporters = ManualConfig.CreateEmpty()
                .AddColumnProvider(DefaultColumnProviders.Instance)
                .AddLogger(ConsoleLogger.Default);

and pass it to BenchmarkRunner|Switcher