Benchmarkdotnet: RPlotExporter hanging

[bonesoul]

// * Export *
  BenchmarkDotNet.Artifacts\results\Hypepool.Benchmarks.Utils.Helpers.Time.Fastclock-report.csv
  BenchmarkDotNet.Artifacts\results\Hypepool.Benchmarks.Utils.Helpers.Time.Fastclock-report-github.md
  BenchmarkDotNet.Artifacts\results\Hypepool.Benchmarks.Utils.Helpers.Time.Fastclock-report.html
  BenchmarkDotNet.Artifacts\results\Hypepool.Benchmarks.Utils.Helpers.Time.Fastclock-measurements.csv
  BenchmarkDotNet.Artifacts\results\Hypepool.Benchmarks.Utils.Helpers.Time.Fastclock-measurements.csv
  BuildPlots.R

and then it hangs?

[AndreyAkinshin]

@bonesoul, which version of BenchmarkDotNet do you use? And which OS?

[bonesoul]

0.10.12 from nuget.
os: win10-x64.

[adamsitnik]

Hi @bonesoul

I just faced the same problem.

It turns out that I had a fresh R install and R needed to download all the required packages first.

Workaround: Find your location of the Rscript.exe file and provide it two arguments: path to BuildPlots.R file from BenchmarkDotNet.Artifacts\results and some *.measurements.csv file from the same folder.

Example:
C:\Projects\Benchmarks\Benchmarks\BenchmarkDotNet.Artifacts\results>"C:\Program Files\R\R-3.4.3\bin\Rscript.exe" BuildPlots.R Json_ToString_LoginViewModel_-measurements.csv

it's going to download all the packages and export the plots. Next time you won't need that, it will work OOB with BenchmarkDotNet.

[codefox42]

Today I also faced this problem (BechnmarkDotNet-0.11.1, Win10-x64, R-3.4.1). In my case just running Rscript.exe BuildPlots.R ...measurement.csv for some reason did not change the hanging behavior. After performing the following steps (with a benchmark run in between), exporting does not hang any more:

• Update R-3.4.1 to 3.5.1
• Remove temporary R-Folders (%TEMP%\Rtmp*)
• Remove library folder (%USERPROFILE%\Documents\R
• Run Rscript.exe BuildPlots.R ...measurement.csv by hand

I guess, the last two steps should be sufficient (which I cannot verify at the moment).

[m-mccormick]

I ran into this issue using R 3.5.2 and BenchmarkDotNet 0.11.3. When running the benchmark executable, it got hung up when building the R script with the console visible. The litmus test for R script completion I'm using is whether I can open the PDF generated by the R script. When it hung and I couldn't open the PDF, I hit "Ctrl+C" with the console active, which killed the benchmark executable, but the R script became unhung and finished the operation.

To diagnose, I created a benchmark (lots of parameters) which creates 312 images, and therefore a lot of stdout from the R process. It appears that stdout redirection is what causes the hang, or something related to the stdout stream. After successfully running my benchmark application to generate the csv and R script, I commented out the benchmark and ran the following method instead.

Trial Number - Result
0 - Runs the whole way through without issue
1 - Generates 14 images than hangs. Pdf won't open/produces error.
2 - Generates 14 images than hangs. Pdf won't open/produces error.

private static bool RunRscript(string resultsDirectory, int trialNumber)
        {
            string rPath = @"C:\Program Files\R\R-3.5.2\bin\Rscript.exe";
            string scriptPath = Path.GetFullPath(resultsDirectory + "/BuildPlots.R");

            ProcessStartInfo psi = new ProcessStartInfo(rPath, scriptPath)
                                       {
                                           WorkingDirectory = resultsDirectory,
                                           UseShellExecute = false
                                       };

            if (trialNumber > 0)
            {
                psi.RedirectStandardOutput = true;
            }

            Process process = new Process
                                  {
                                      StartInfo = psi
                                  };

            StringBuilder sb = new StringBuilder();
            if (trialNumber == 1)
            {
                process.OutputDataReceived += delegate(object sender, DataReceivedEventArgs args)
                        {
                            sb.Append(args.Data);
                        };
            }

            if (!process.Start())
            {
                throw new Exception("Couldn't Start R");
            }

            process.WaitForExit();

            if (psi.RedirectStandardOutput && trialNumber == 2)
            {
                sb.Append(process.StandardOutput.ReadToEnd());
            }

            if (process.ExitCode != 0)
            {
                return false;
            }

            return true;
        }

[m-mccormick]

I modified my test to better reflect public IEnumerable<string> ExportToFiles(Summary summary, ILogger consoleLogger) in RPlotExporter.cs. Also, I forgot process.BeginOutputReadLine(); and process.BeginErrorReadLine(); on trial 1 in the previous post. New Results:

Trial Number - Result
0 - Same, Runs the whole way through without issue, but obviously there is nothing to write to the logs.
1 - Runs the whole way through without issue, and stdout and stderr are intact for use by the ILogger
2 - Generates 14 images than hangs. Pdf won't open/produces error. Ctrl+C kills benchmark process, which allows the R script to continue running to completion, then the Pdf opens without issue.

@AndreyAkinshin
It appears that event driven stream reading of stdout and stderr should fix this bug.

Code:

/// <summary>
/// Run an R script
/// </summary>
/// <param name="resultsDirectory">The BenchmarkDotNet.Artifacts/results directory</param>
/// <param name="trialNumber">
/// 0 - Run with no stdout or stderr
/// 1 - Run with event-driven stdout and stderr
/// 2 - Run with alternating synchronous read of stdout and stderr
/// </param>
/// <returns>Whether the script was run successfully</returns>
private static void RunRscript(string resultsDirectory, int trialNumber)
{
    string rPath = @"C:\Program Files\R\R-3.5.2\bin\Rscript.exe";
    string scriptPath = Path.GetFullPath(resultsDirectory + "/BuildPlots.R");

    ProcessStartInfo start = new ProcessStartInfo
                                 {
                                     FileName = rPath,
                                     Arguments = scriptPath,
                                     UseShellExecute = false,
                                     CreateNoWindow = true,
                                     WorkingDirectory = resultsDirectory,
                                 };

    if (trialNumber > 0)
    {
        start.RedirectStandardOutput = true;
        start.RedirectStandardError = true;
    }

    StringBuilder sbStdOut = new StringBuilder();
    StringBuilder sbStdErr = new StringBuilder();
    using (var process = Process.Start(start))
    {
        if (process == null)
            throw new Exception("Process Null");

        if (trialNumber == 1)
        {
            // Works like a charm
            process.OutputDataReceived += delegate(object sender, DataReceivedEventArgs args)
                {
                    lock (sbStdOut)
                        if (!string.IsNullOrWhiteSpace(args.Data))
                            sbStdOut.Append(args.Data);

                };
            process.ErrorDataReceived += delegate(object sender, DataReceivedEventArgs args)
                {
                    lock (sbStdErr)
                        if (!string.IsNullOrWhiteSpace(args.Data))
                            sbStdErr.Append(args.Data);
                };

            process.BeginOutputReadLine();
            process.BeginErrorReadLine();
        }

        if (trialNumber == 2)
        {
            // Hangs Up
            // sbStdOut.Append(process.StandardOutput.ReadToEnd());
            // sbStdErr.Append(process.StandardError.ReadToEnd());

            // Alternate Approach, still hangs
            while (!process.HasExited)
            {
                if (process.StandardOutput.Peek() >= 0)
                    sbStdOut.Append(process.StandardOutput.ReadLine());

                if (process.StandardError.Peek() >= 0)
                    sbStdErr.Append(process.StandardError.ReadLine());
            }
        }

        if (trialNumber <= 1)
        {
            process.WaitForExit();
        }

        if (process.ExitCode != 0)
            throw new Exception("Error Running R Script");

        // To ILogger...
        Console.WriteLine("\n\nStdOut: " + sbStdOut);
        Console.WriteLine("\n\nStdIn: " + sbStdErr);
    }
}

[mstaessen]

I have the impression that the R script is waiting for user input at some point. If you Ctrl+C in the terminal, R stops hanging and all the plots start appearing...

[tmulkern]

I have the impression that the R script is waiting for user input at some point. If you Ctrl+C in the terminal, R stops hanging and all the plots start appearing...

@mstaessen I can confirm that is the case. It hung for me and Ctrl+C stopped RScript.exe and finished the plots

[ccarrero301]

It happened to me the first benchmark run, after that seems to be working. Any permanent solution for this?