Node-fs-extra: Documentation for copy() options.filter function

[stephen-last]

I'm not sure how to use the options.filter function with copy().

The docs say:

Function to filter copied files. Return true to include, false to exclude.

I need to copy a folder and sub files/folders, however I need to exclude some from the copy (I'm copying a node project folder and need to exclude the node_modules folder within). The above makes it sound like this function can used but I'm not sure how.

Can this function be used for what I need..?

[RyanZim]

Quick example:

fs.copy(src, dest, {
  filter: function (path) {
    return path.indexOf('node_modules') > -1
  }
}, callback)

We should add an example to the docs.

[pwang2]

@stephen-last You may have the same puzzle as I had. When copying directory and use the filter to include/exclude files in subdirectory, it actually filters from the root first. If the root folder does not match the filter function, it did not go to the subdirectory at all. To make it work as expected, I use this workaround. here if (fs.lstatSync(n).isDirectory()) {return true} is needed to bypass the root folder to have chance to match sub directories and files.

├── bar
│   └── baz
│       └── file1.txt
└── foo
    ├── baz
    │   └── file1.txt
    └── mie.js

var fs = require('fs-extra');
fs.copy('./src', './dist', {
        clobber: true,
        filter: n => {
             if (fs.lstatSync(n).isDirectory()) {
                return true;
            }
            var result = /\/baz\//.test(n);
            console.log(result ? 'copied' : 'skipped', n);
            return result;
        }
    },
    () => console.log('done')
);

This gives output like this.

➜ test node index.js
skipped /private/tmp/test/src/foo/mie.js
copied /private/tmp/test/src/foo/baz/file1.txt
copied /private/tmp/test/src/bar/baz/file1.txt
done

[stephen-last]

This seems to work for me, it copies everything apart from path's containing node_modules.

import Promise from 'bluebird'
import fse from 'fs-extra'

const fs = Promise.promisifyAll(fse)

const copyFrom = '...'
const copyTo = '...'

fs.copyAsync(copyFrom, copyTo, {
  filter: path => {
    console.log('path ===', path)
    return !(path.indexOf('node_modules') > -1)
  }
}).then(() => {
  console.log('done')
}).catch(err => {
  console.error(err)
})

@pwang2 You have clobber: true... This isn't in the docs either. Where did you find that..?

[RyanZim]

@stephen-last clobber was renamed to overwrite in v2. Technically, clobber still works, but I that will probably be removed in a later major release.

https://github.com/jprichardson/node-fs-extra/blob/master/docs/copy.md#options

[stephen-last]

@RyanZim I've forked this repo and created a pull request with an update for the docs.

I've not done this before, is that how I suggest an update..?

[pwang2]

@RyanZim & @stephen-last , does it work for you without the lsstat isDirectory workaround? In my test, it only matched the root folder and return false in the filter and quit. Did I miss anything here? Thanks..

skipped /private/tmp/test/src
done

[stephen-last]

@pwang2 Yes it works without isDirectory(). We're doing different things. I'm trying to miss out any paths with node_modules in, and you're trying to only include paths with baz in (I think!). Your code is just copying all folders/subfolders because they're missed out from your baz test. Your root folder is /private/tmp/test/src, which doesn't contain baz and so is missed out (which means all sub files and folders are not copied).

[pwang2]

Emmm, you are right but it sound weird/anti-intuitive to me. When we say a
filter, it is more like a opt in function to me. Here, it only works in an
opt-out way.

On Tue, Feb 7, 2017 at 3:07 AM Stephen Last [e-mail]
wrote:

@pwang2 https://github.com/pwang2 Yes it works without isDirectory().
We're doing different things. I'm trying to miss out any paths with
node_modules in, and you're trying to only include paths with baz in (I
think!). Your code is just copying all folders/subfolders because they're
missed out from your baz test. Your root folder is /private/tmp/test/src,
which doesn't contain baz and so is missed out (which means all sub files
and folders are not copied).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jprichardson/node-fs-extra/issues/350#issuecomment-277940549,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABhKzsKcF44JLKUhSPHpghNg0p2hCCoUks5raDRRgaJpZM4L2kme
.

>

Peng

[stephen-last]

@pwang2 You return true to include or false to exclude. It's up to you whether to make it opt-in or opt-out using your logic. The filter() function doesn't prescribe a way of filtering, that's up to you. Your code is opt-in (only paths with baz), mine is opt-out (all paths without node_modules).

[pwang2]

@stephen-last, as shown above, we still need the if (fs.lstatSync(n).isDirectory()) to make opt-in work. It would be better to function without the isDirectory check and just make it like a glob matching as below.

 find . -path './**/*baz*' -print0 |  xargs -0 ls

I think it will be less confusing, please share your thoughts.

[stephen-last]

@pwang2 I don't know much about glob matching so I can't comment on that.

Your isDirectory() call is only missing out directories from your baz check. You could test for the root folder instead:

const options = {
  filter: path => {
    // this is the root path
    if (path === '/private/tmp/test/src') { return true }
    // do your normal checking (on files *and* folders)
    const isBaz = (path.indexOf('baz') > -1)
    console.log(isBaz ? 'copied' : 'skipped', path)
    return isBaz
  }
}

[RyanZim]

FWIW, glob support ain't happening anytime soon; maybe sometime in the distant future.

[httpete]

this bit me when I upgraded to 2.0.0. If I didn't use the check for isDirectory. This worked fine before the upgrade to 2.

  filter: n => {
         if (fs.lstatSync(n).isDirectory()) {
            return true;
        }
        var result = /\/baz\//.test(n);
        console.log(result ? 'copied' : 'skipped', n);
        return result;
    }

[pwang2]

This could be easily fixed if people agree on.

function startCopy (source) {
    started++
    if (filter) {
      if (filter instanceof RegExp) {
        console.warn('Warning: fs-extra: Passing a RegExp filter is deprecated, use a function')
        if (!filter.test(source)) {
          return doneOne(true)
        }
      } else if (typeof filter === 'function') {
        if (!filter(source)) {
          return doneOne(true)
        }
      }
    }
    return getStats(source)
  }

In source code above, the filter is skipping(call doneOne(true)) even for root entry which definitely return false for sub directory/files matching , so unless we use an opt-out match, it will not work as expected.

This should behave like

cp -r test/*  dist

what we need as copy here is a glob, so the source should not be a dirname and use a filter to match child.

The exclusive filter to me is like another alternative in POSIX cmd.

# need to open extglob in most platforms
cp -r test/!(node_modules)/*  target  

I think it would be better to make it behave more like POSIX alternative here like other utils in node fs module do. But I could feel that it should be a reason that we miss fs.cp in node core fs module

[manidlou]

Just a quick note, isDirectory() is not always the correct workaround. Imagine if you have something like this:

├── bar
│   └── baz
│       └── file1.txt
└── foo
|    ├── baz
|    │   └── file1.txt
|    └── mie.js
|___ qux
    |__ file3.md

by doing this:

const fs = require('fs-extra')

const fn = (src, dest) => {
  if (fs.lstatSync(src).isDirectory()) return true
  return /\/baz\//.test(src) // or src.indexOf('baz') > -1
}

fs.copy('src', 'dest', {filter: fn}, err => {
  if (err) return console.log(err)
  console.log('done.')
})

You end up having an empty qux directory in dest. So, it mostly depends on the src directory structure and the use case.

[guohuihot]

var glob = require('glob');

fs.removeGlobSync = function(g) {
        var files = glob.sync(g);
        files.forEach(function(file) {
            fs.removeSync(file);
        });
    }

fs.copyGlobSync = function(g) {
        var files = glob.sync(g);
        files.forEach(function(file) {
            fs.copySync(file);
        });
    }

[manidlou]

I think the filter option is sufficiently clear now in the docs. Isn't it? I am not sure if we need to keep this open! @RyanZim?

[RyanZim]

Probably right; closing.

[bytrangle]

I may be missing something, but where is the documentation for using the filter function in copy and copySync? Both docs only say:

“Function to filter copied files. Return true to include, false to exclude. Can also return a Promise that resolves to true or false (or pass in an async function).”

It wasn't clear to me how to use the filter function until I googled and bumped into this issue.

[BassOfBass]

Yeah, and there is no explanation for src and dist arguments. At first I assumed they were related to the original paths in the copy() function, but apparently it's not the case and src is just a path of the copied file/sub-folder. What's dist then?