Node: Add fs.copyFile for copying files, using uv_fs_copyfile

Created on 18 Aug 2017  路  9Comments  路  Source: nodejs/node

A new uv_fs_copyfile function recently landed in libuv in order to allow more efficient copying of files (in the future, it could allow copy-on-write semantics on file systems that support it). A copyFile function that uses this should be added to Node.js. 馃槂

Depends on upgrade to libuv 1.14.0 (#14866)

feature request fs libuv

Most helpful comment

Could someone detail the advantages of uv_fs_copyfile?

Two main advantages:

  1. It's significantly faster than a JS implementation, as it uses native system calls where available (eg. CopyFile on Windows, copyfile(3) on Mac OS). See some benchmark results on https://github.com/yarnpkg/yarn/pull/3290 and https://github.com/yarnpkg/yarn/pull/2960 for example.
  2. It allows us to take advantage of copy-on-write semantics on file systems that support it, such as ZFS, BTRFS, ReFS, APFS. Copy-on-write means that creating a copy of a file reuses the same data on disk, similar to a hardlink. It uses very very little additional disk space, until the file is modified (at which point it's actually physically copied). This is hugely beneficial for apps that deal with lots of file copies, like Yarn and npm. It allows the benefits of hardlinks/symlinks (very fast installation, reusing the same data on the disk) without any of the disadvantages (changing one of the copies does not modify any of the other ones).

Copy-on-write needs native system calls to function, as the hacky mechanism used in Node.js projects at the moment (read the input file into a buffer and write it to the new location) is not recognised as a file copy by the OS. The file system just sees it as creation of a brand new file.

All 9 comments

I'll be working on this once libuv can be updated. Right now, it's blocked on a broken test.

Thank you @cjihrig! I just wanted to create an issue for tracking purposes 馃槂

Could someone detail the advantages of uv_fs_copyfile?

Do note that we make no guarentees to expose every libuv API. 馃槈

Could someone detail the advantages of uv_fs_copyfile?

Two main advantages:

  1. It's significantly faster than a JS implementation, as it uses native system calls where available (eg. CopyFile on Windows, copyfile(3) on Mac OS). See some benchmark results on https://github.com/yarnpkg/yarn/pull/3290 and https://github.com/yarnpkg/yarn/pull/2960 for example.
  2. It allows us to take advantage of copy-on-write semantics on file systems that support it, such as ZFS, BTRFS, ReFS, APFS. Copy-on-write means that creating a copy of a file reuses the same data on disk, similar to a hardlink. It uses very very little additional disk space, until the file is modified (at which point it's actually physically copied). This is hugely beneficial for apps that deal with lots of file copies, like Yarn and npm. It allows the benefits of hardlinks/symlinks (very fast installation, reusing the same data on the disk) without any of the disadvantages (changing one of the copies does not modify any of the other ones).

Copy-on-write needs native system calls to function, as the hacky mechanism used in Node.js projects at the moment (read the input file into a buffer and write it to the new location) is not recognised as a file copy by the OS. The file system just sees it as creation of a brand new file.

One thing to keep an eye out for. Replacing a JS implementation with a native API for file system related stuff will end up changing the underlying syscall failures if anything goes wrong. We saw this happen with realpath, and things got weird quickly.

This is in no way saying we shouldn't do this, but we should be pretty thorough with smoke testing if we do decide to change things. Either way it will likely be semver major, even if not obviously so

We aren't replacing anything here though. This would be purely semver minor for core. The applications using it can pick their own level of semver.

. Replacing a JS implementation with a native API

There is no core JS implementation at the moment. This would add a brand new function, so there's no breaking change. Currently, apps are implementing their own JS versions of it.

Existing https://github.com/nodejs/node/issues/12902 requests the same feature.

I am in favor of implementing this on core :+1:

Was this page helpful?
0 / 5 - 0 ratings