Roslyn: Microsoft.CodeAnalysis.CSharp.Scripting causes Too many open files when ran multiple times

[jasekiw]

Version Used:
3.4.0

Steps to Reproduce:

1. Test on a linux server or the server I was using CentOS Linux release 7.6.1810 (Core)
2. Create an asp.net core project
3. Add a call to var objectResult = await CSharpScript.EvaluateAsync("return 1"); in a controller.
4. Call this route over and over again until the max limit is reached.

Expected Behavior:

For the scripting API to close files after it is done with them.

Actual Behavior:

/dev/zero is opened until the server reaches it's max limit of open files and the following error occurs and asp.net core locks up not able to accept any more requests.

Everything with namespace UptimeRMX is my project.

System.IO.IOException: Too many open files
    at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
    at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
    at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
    at System.IO.File.OpenRead(String path)
    at Roslyn.Utilities.FileUtilities.OpenFileStream(String path)
    at Microsoft.CodeAnalysis.MetadataReference.CreateFromAssemblyInternal(Assembly assembly, MetadataReferenceProperties properties, DocumentationProvider documentation)
    at Microsoft.CodeAnalysis.Scripting.Script.GetReferencesForCompilation(CommonMessageProvider messageProvider, DiagnosticBag diagnostics, MetadataReference languageRuntimeReferenceOpt)
    at Microsoft.CodeAnalysis.CSharp.Scripting.CSharpScriptCompiler.CreateSubmission(Script script)
    at Microsoft.CodeAnalysis.Scripting.Script.GetCompilation()
    at Microsoft.CodeAnalysis.Scripting.Script`1.GetExecutor(CancellationToken cancellationToken)
    at Microsoft.CodeAnalysis.Scripting.Script`1.RunAsync(Object globals, Func`2 catchException, CancellationToken cancellationToken)
    at UptimeRMX.Api.DataProcessor.CalculationProcessor.ProcessCalculation(String code, List`1 variables)

Upon inspection of the files opened, I see most of the files are opened to /dev/zero and the number increases every time the above command is ran.

I am reporting on how many open files there are to /dev/zero with the following command:

lsof -p <pid-of-process>| grep /dev/zero | wc -l

You can get the problem to happen very quickly if you execute it in a loop

        [AllowAnonymous]
        [HttpPost("TestDevZeroBug")]
        public async Task<IActionResult> TestDevZeroBug()
        {
            object result = 0;
            for (int i = 0; i < 4000; i++)
            {
                result = await CSharpScript.EvaluateAsync("return 1;");
            }
            return Ok(new CalculationTestResult {Output = JsonConvert.SerializeObject(result)});
        }

[jinujoseph]

cc @tmat

[ReginaldBull]

Experiencing the same issue on Ubuntu 18.04 LTS

[ReginaldBull]

Hello,
Any news about this issue?

[jasekiw]

@ReginaldBull I am pretty sure this issue is related to

https://github.com/dotnet/roslyn/issues/41722
https://github.com/dotnet/roslyn/issues/41348

@tmat mentioned in those issues that assembly unloading is not supported at the moment which I believe might be causing the opened files.

If your implementation resuses the same code a lot, You can reduce the affects quite a bit by building the script then executing it multiple times.

https://github.com/dotnet/roslyn/wiki/Scripting-API-Samples#multi

I was able to get drastically better results by doing this but your mileage will vary by how much reuse of the same code you use.

[dave-yotta]

I've tried manually loading and unloading the compilation via CSharpCompilation and AssemblyLoadContext.Unload, it does behave better, but still eventually hangs or dies with "Too many open files".

https://github.com/dotnet/roslyn/issues/49282

[dave-yotta]

Update: I found my mistake with AssemblyLoadContext.

If you want to get around these leaks you can follow roughly what I was doing in this test project here: https://github.com/dave-yotta/roslyn-assemblyunload