Runtime: Provide a File.CopyAsync method

[davidmatson]

Related to dotnet/runtime#20695.

File.Copy currently blocks the calling thread. As mentioned in the related issue, .NET alternatives lose significant performance optimizations done within the CopyFile* APIs. It would be great if .NET had a good, general purpose, async file copy method. Doing so well (fast) probably requires a new Win32 API that support something like overlapped I/O to avoid blocking the calling thread.

Some possible overloads:

• Task CopyAsync(string sourceFileName, string destFileName);
• Task CopyAsync(string sourceFileName, string destFileName, CancellationToken cancellationToken);
• Task CopyAsync(string sourceFileName, string destFileName, IProgress<CopyProgressInfo> progress, CancellationToken cancellationToken);

It probably makes sense to provide an async version of File.Move at the same time, which also likely requires new Win32 API support.

[davidmatson]

Suggested labels (don't think I have permissions to assign):

• area-System.IO
• enhancement

[SamuelEnglard]

I don't think new Win32 APIs would be needed, though I full admit they would make it much easier. Using the CopyProgressRoutine callback should be enough.

[davidmatson]

How would the method be made async without a new Win32 API? The current CopyFile* APIs all block the calling thread until the copy completes, right?

[SamuelEnglard]

I thought there was a way to have them return early but on looking again seems there isn't. I very much suspect there is a way to do this, just maybe not obvious.

[SamuelEnglard]

A thought: While not great one solution would be to queue requests to the same disk (or at least volume) up on a dedicated thread similar to how timers work.

[JeremyKuhne]

I don't know of any existing async Win32 APIs. I'm fine with creating an Async wrapper that wraps the calls in a Task- I'm not sure you need anything more complicated than that?

[jnm2]

I don't know of any existing async Win32 APIs. I'm fine with creating an Async wrapper that wraps the calls in a Task- I'm not sure you need anything more complicated than that?

If there is no async API, I'd prefer a synchronous method with cancellation and progress support. People can wrap that in Task.Run and they won't be under the impression that it's not hogging a thread that way.

I'm curious what @StephenCleary thinks.

[davidmatson]

@jnm2 : I agree; I've seen folks call it an anti-pattern to wrap something in a task when it isn't really async under the hood.

I'm also interested in exploring adding a Win 32 API with async support, though that would likely require business justification, such as ".NET wants to use this API." Let me know if there's interest in pursuing that option.

[StephenCleary]

I agree in general - Task.Run wrappers over non-asynchronous APIs are not recommended. However, I'd also say that it's up to each platform whether to do this. For example, Node.js keeps a threadpool around specifically to block threads on APIs that should be asynchronous but aren't. Similarly, I believe WinRT also does this for some APIs for consistency. In particular, there's no Win32 notion of an asynchronous file-open API, but all file-opens on WinRT are async, so I assume they use the thread pool (either that, or they actually are synchronous).

It looks like CopyFileEx does support both progress reporting and cancellation. The p/Invoke would be convoluted, with a volatile BOOL for cancellation (blech) as well as the reverse-p/Invoke for progress reporting. Convoluted, but doable.

However, a synchronous CopyFile with progress reporting and cancellation is really just catering to the limitations of a particular platform (Win32). If other platforms (or future platforms) support asynchronous file copies, IMO it would be preferable to have netstandard define an asynchronous API, which is implemented using Task.Run on (current) Win32 systems due to platform limitations.

[danmosemsft]

@rainersigwald would MSBuild take advantage of this? I notice some limiting in your parallelized Copy due to concerns about the threadpool. Possibly with this you could just fire them all off together.

cc @JeremyKuhne

[https://github.com/Microsoft/msbuild/pull/3331]

[stephentoub]

would MSBuild take advantage of this? I notice some limiting in your parallelized Copy due to concerns about the threadpool. Possibly with this you could just fire them all off together.

I'm skeptical we'd see real benefits from this. To my knowledge there aren't any Win32 APIs for opening a file asynchronously nor for copying a file asynchronously (e.g. a single overlapped operation surfaced through Win32 for the whole copy). That means we'd end up doing something like what we do in our FileStream.CopyToAsync implementation, which is manually do each read and write using overlapped I/O. We could improve on it a bit, taking advantage of the fact that we could avoid the FileStream abstraction, but we'd still be looking at difference between making a single Win32 call to CopyFileEx vs doing the whole copy ourselves.

Just as a little example of this, I generated a 6GB file and then copied it using File.Copy and FileStream.CopyToAsync, measuring each:
```C#
using System;
using System.Diagnostics;
using System.IO;

class Program
{
static void Main()
{
string src = @"src.dat";
string dst = @"dst.dat";

    //byte[] buffer = new byte[6 * 1024 * 1024];
    //new Random().NextBytes(buffer);
    //using (var f = File.OpenWrite(src))
    //{
    //    for (int i = 0; i < 1024; i++)
    //        f.Write(buffer, 0, buffer.Length);
    //}

    var sw = new Stopwatch();
    while (true)
    {
        File.Delete(dst);
        sw.Restart();
        File.Copy(src, dst);
        sw.Stop();
        Console.WriteLine("File.Copy: " + sw.Elapsed.TotalSeconds);

        File.Delete(dst);
        sw.Restart();
        using (var srcStream = new FileStream(src, FileMode.Open, FileAccess.Read, FileShare.Read, 0x1000, useAsync: true))
        using (var dstStream = new FileStream(dst, FileMode.Create, FileAccess.Write, FileShare.Write, 0x1000, useAsync: true))
        {
            srcStream.CopyToAsync(dstStream).GetAwaiter().GetResult();
        }
        sw.Stop();
        Console.WriteLine("Async    :" + sw.Elapsed.TotalSeconds);

        Console.WriteLine();
    }
}

}

On my machine I get output like this:

File.Copy: 5.5099244
Async : 20.1811448

File.Copy: 5.2296971
Async : 20.509725

File.Copy: 5.6288642
Async : 19.7780366
```
From a scalability perspective, it's possible there'd be some wins to be had if the copies were happening across a network or something, such that we could avoid burning a bunch of threads doing a ton of copies in parallel, but that's a huge gap to be recovered.

[rainersigwald]

I wouldn't object to using something like this in MSBuild, though we've run into trouble in the past when using the threadpool for anything in VS, and having our own throttling mechanism helps with that. Copy-task perf is not currently on my radar, though, so the wins would have to be pretty clear to take the change over the now-fairly-well-established existing code.

I also would prefer to stick as close to a single-syscall implementation of Copy as possible, since that has the advantage of preserving OS behavior with respect to all the other aspects of files (modified times, other metadata, maybe extended attributes on macOS?). So moving to async streaming has another strike against it beyond the perf @stephentoub mentions.

[StephenCleary]

I'm not convinced this would be good for MSBuild. It would only make sense for this use case IMO if the file copies were blocking something else from happening (i.e., blocking a thread that could be doing something completely different). Parallel file copies would rarely make sense for MSBuild; it generally runs on local files (not network), and any parallelizing would usually be applied over files in the same directory tree. I suspect that kind of parallelism is more likely to be harmful than helpful, even if we assume all devs are on SSD rather than spinning rust (a rather optimistic assumption).

The question of whether the BCL should have an asynchronous File.CopyAsync is different. There's a decent case to be made for that.

[rainersigwald]

Parallel file copies would rarely make sense for MSBuild

This is empirically not the case; see the stats that convinced us to take the current parallelized version: https://github.com/Microsoft/msbuild/pull/3331/files#diff-8e199870eb02037007e3e9cb57963f23R25

[danmosemsft]

Ah, I just assumed there was a built in Win32 CopyFileAsync but I did not check. I agree that makes this API of limited value wrt perf and atomicity.

@JeremyKuhne perhaps we should close this?

[davidmatson]

I'm looking for whether there would be justification to add an async version of the Win32 CopyFile API - if we added this in Windows, would it be used in .NET?

[JeremyKuhne]

perhaps we should close this?

I'm reading that MSBuild saw benefits to doing async over sync copies. Am I reading that right @rainersigwald?

I think we'd be better off starting with adding progress support, although I haven't looked into what that entails with Unix. https://github.com/dotnet/corefx/issues/17306#issuecomment-295878901

if we added this in Windows, would it be used in .NET?

If it could make things significantly better for developers, sure. We'd have to pull together a rational Unix implemenation though. That puts extra weight on this.

[rainersigwald]

I'm reading that MSBuild saw benefits to doing async over sync copies. Am I reading that right @rainersigwald?

Yes, with caveats:

• It wasn't in a heavily async environment; we just started some copier threads in a process that's more or less single threaded
• There was an upper bound to how many copies in flight was helpful, and it varied based on disk type (see stats I linked above)

[JeremyKuhne]

The right way forward for this is implementing dotnet/runtime#20695.