Runtime: `Directory.EnumerateFiles(dir, pat)` is sometimes 45-50x slower on .NET Core 3.0 compared to .NET Framework 4.8 / .NET Core 2.0

[0xd4d]

Directory.EnumerateFiles(dir, pat) is sometimes 45-50x slower on .NET Core 3.0 compared to .NET Framework 4.8 / .NET Core 2.0

Discovered by @iSazonov, see https://github.com/PowerShell/PowerShell/issues/6577#issuecomment-543194018

Related:

• https://github.com/dotnet/corefx/pull/25426
• https://github.com/dotnet/corefx/pull/26806
• https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-2-1/

Methods:

• PS1(): 2000 x Directory.EnumerateFiles(@"C:\WINDOWS\system32\", "ping.*")
• Blog(): 2000 x Directory.EnumerateFiles(@"d:\repos\corefx\src\System.IO.FileSystem\", "*.cs", SearchOption.AllDirectories)

.NET | PS1() | Blog()
-----|-------|-------
.NET Framework 4.8 | 78 ms | 4393 ms
.NET Core 1.0 | 93 ms | 5298 ms
.NET Core 1.1 | 91 ms | 4712 ms
.NET Core 2.0 | 76 ms | 4729 ms
.NET Core 2.1 | 4314 ms | 1585 ms
.NET Core 2.2 | 4286 ms | 1599 ms
.NET Core 3.0 | 3788 ms | 1538 ms
.NET Core 3.1 | 3753 ms | 1538 ms

You can also test this PowerShell command

Measure-Command { for ($i=0; $i -lt 2000; $i++) { Get-Command ping > $null } }

PS | tfm | Time (s)
---|-----|---------
PS 5.1 | net48 | 0.5251987
PS 7.0 | netcoreapp3.0 | 4.1898758

Most expensive calls:

.NET Framework 4.8 and .NET Core 2.0

Name                                                                                            Inc %   Inc
 consoleapp1!ConsoleApp1.Program.PS1(class System.String)                                         2.2    90
+ mscorlib.ni!Directory.EnumerateFiles                                                            1.5    62
|+ mscorlib.ni!System.IO.FileSystemEnumerableIterator`1[System.__Canon]..ctor(...)                1.5    61
||+ mscorlib.ni!System.IO.FileSystemEnumerableIterator`1[System.__Canon].CommonInit()             1.2    49
|| + mscorlib.ni!DomainNeutralILStubClass.IL_STUB_PInvoke(System.String, WIN32_FIND_DATA ByRef)   1.2    48
||  + kernelbase!FindFirstFileW                                                                   1.2    47
||   + kernelbase!FindFirstFileExW                                                                1.2    47

.NET Core 3.1 (3.1.100-preview1-014459)

Name                                                                                                     Inc %      Inc
 consoleapp1!ConsoleApp1.Program.PS1(class System.String)                                                 62.6    2,884
+ system.io.filesystem.il!System.IO.Enumeration.FileSystemEnumerator`1[System.__Canon].MoveNext()         53.8    2,479
|+ system.io.filesystem.il!System.IO.Enumeration.FileSystemEnumerator`1[System.__Canon].FindNextEntry()   47.2    2,175
||+ system.io.filesystem.il!dynamicClass.IL_STUB_PInvoke(...)                                             45.4    2,093
|||+ ntdll!NtQueryDirectoryFile                                                                           45.3    2,086

Repro

• Update the path to corefx repo in the source code below
• Compile the code below
• Run the executable on Windows

```C#
using System;
using System.Diagnostics;
using System.IO;
using System.Runtime.CompilerServices;

namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
const string psPattern = "ping.";
const string blogPattern = ".cs";
for (int i = 0; i < 100; i++) {
PS1(psPattern);
Blog(blogPattern);
}
var swPS1 = Stopwatch.StartNew();
for (int i = 0; i < 2000; i++)
PS1(psPattern);
swPS1.Stop();
var swBlog = Stopwatch.StartNew();
for (int i = 0; i < 2000; i++)
Blog(blogPattern);
swBlog.Stop();
Console.WriteLine($"{IntPtr.Size * 8}-bit");

if NETCOREAPP1_0

        Console.WriteLine("netcoreapp1.0");

elif NETCOREAPP1_1

        Console.WriteLine("netcoreapp1.1");

else

        Console.WriteLine(typeof(object).Assembly.Location);

endif

        Console.WriteLine($"PS1 : {swPS1.ElapsedMilliseconds} ms");
        Console.WriteLine($"Blog: {swBlog.ElapsedMilliseconds} ms");
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Blog(string pattern)
    {
        foreach (string path in Directory.EnumerateFiles(@"d:\repos\corefx\src\System.IO.FileSystem\", pattern, SearchOption.AllDirectories))
        {
            Use(path);
        }
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void PS1(string pattern)
    {
        var directories = new string[]
        {
            @"C:\WINDOWS\system32\",
        };
        foreach (var directory in directories)
        {
            foreach (var f in Directory.EnumerateFiles(directory, pattern))
            {
                Use(f);
            }
        }
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    static void Use(string s) { }
}

}

```csproj
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp3.1;netcoreapp3.0;netcoreapp2.2;netcoreapp2.1;netcoreapp2.0;netcoreapp1.1;netcoreapp1.0;net48</TargetFrameworks>
  </PropertyGroup>

</Project>

dotnet build -c Release -f net48
dotnet publish -c Release -f netcoreapp1.0 -r win7-x64 --self-contained
dotnet publish -c Release -f netcoreapp1.1 -r win7-x64 --self-contained
dotnet publish -c Release -f netcoreapp2.0 -r win-x64 --self-contained
dotnet publish -c Release -f netcoreapp2.1 -r win-x64 --self-contained
dotnet publish -c Release -f netcoreapp2.2 -r win-x64 --self-contained
dotnet publish -c Release -f netcoreapp3.0 -r win-x64 --self-contained
dotnet publish -c Release -f netcoreapp3.1 -r win-x64 --self-contained

.\ConsoleApp1\bin\Release\net48\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp1.0\win7-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp1.1\win7-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp2.0\win-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp2.1\win-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp2.2\win-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp3.0\win-x64\publish\ConsoleApp1.exe
echo ''
.\ConsoleApp1\bin\Release\netcoreapp3.1\win-x64\publish\ConsoleApp1.exe
echo ''

[stephentoub]

cc: @jeremykuhne

[danmosemsft]

This is linked to https://github.com/PowerShell/PowerShell/issues/6577 but that seems to be about Linux, which has a signfiicantly different implementation.

[0xd4d]

This is a .NET Core for Windows issue. I haven't tested this on Linux or macOS, though.

[JeremyKuhne]

Initial observations:

In this particular case there are over 4000 files with a single match (ping.exe). It involves a lot more transitions through the interop layer, which is where most of the overhead appears to be. You can increase the buffer to 32k and see the impact (cuts the time roughly in half).

I'm not sure how easily we can automatically optimize for this particular case. We contract to no longer match 8.3 alternate names, which is really tough to express in our API if we try and guess when to pass the filter to NtQueryDirectoryFile. We could add another option to pass in the filter, but we wouldn't want to do that by default. Passing in the filter wouldn't be possible when we add additional match types in the future (say, Regex), of course.

It may also be worth exploring suppressing the GC transition for this call, at least to see the impact.

Whatever we do, it is important to not lose sight of the fact that the overhead in this edge case is less than 2ms. There are significant improvements for more time consuming enumerations that we don't want to lose. We also don't want to lose the cross-plat predictability of removing the 8.3 matching.

[JeremyKuhne]

cc: @carlossanlop

[iSazonov]

@0xd4d Thanks for opening the issue and C# repo!

I measured only on Windows too. As for Unix I guess there is the same because on all platforms a reason of the issue is that now the API reads file attributes even if they are not needed at the moment.
This is not very noticeable with a simple search in a small directory but as mentioned in https://github.com/PowerShell/PowerShell/issues/6577 it becomes significant on slow file systems and scenarios like "find an application by paths from environment PATH variable" - it may already be thousands of files and we read attributes for all them without need, we need to search by name and read attributes only for some result files.

[danmosemsft]

@iSazonov I suggest to open a separate issue for Unix.

[JeremyKuhne]

it becomes significant on slow file systems

Most of the time here is interop and not the file system. Windows itself has to get all of this data and run through it essentially the same way we're doing it. The only real functional difference is they compare the 8.3 names as well in the source data.

There is also significant overhead in just creating the buffer for each directory. That is one reason why the .NET Core code is so much faster doing recursive searches. Perhaps we can add a way to enumerate through sets of directories in the future to help with the PATH scenario you brought up.

[am11]

Is it so that the reason behind directly invoking NTFS API vs. Win32's FindFirstFileW is the support of UNC paths to mitigate PathTooLong problem?

[JeremyKuhne]

Is it so that the reason behind directly invoking NTFS API vs. Win32's FindFirstFileW is the support of UNC paths to mitigate PathTooLong problem?

There were a few reasons:

1. It cuts out the extra layer of translation FindFirst/Next does
2. It allows us to reuse the buffer from directory to directory

Long paths didn't enter into this particular one.

[joshudson]

We also don't want to lose the cross-plat predictability of removing the 8.3 matching.

I found it expedient to just re-do the filter after getting the result back form FindFirstFile/FindNextFile.

[iSazonov]

You can increase the buffer to 32k and see the impact (cuts the time roughly in half).

I tried buffer size from 1 Kb to 512 Kb and don't see difference (on SSD disk).

[JeremyKuhne]

Triage: For cases like this and also some scenarios where matching is being done over UNC shares on huge sets, using the underlying matching in Windows would potentially be beneficial. On Unix it wouldn't be as matching is done in a separate API call. Adding a "System.IO.MatchType.UseSystemMatch" option is possible, but has a lot of complexity to it, including matching 8.3 filenames on Windows which aren't always generated. It also would generate differences on Unix as Unix match APIs don't have Win32 matching semantics (notably collapsing of ? to the last period in the match specifier, and the collapse of the period itself). Perhaps we could call it UseSystemMatchOnWindows? :)

We can't really automatically fall back to the OS match because of the 8.3 caveat.