Azure-storage-azcopy: Exclude a directory path and all files under it

[benc-uk]

I wish to exclude a entire file path from the sync, e.g. node_modules/ folder which I do not need and can be very large.

I have tried every combination in the exclude that I can think of and none work, e.g.

--exclude="node_modules"
--exclude="*node_modules*"
--exclude="node_modules/*"
--exclude="node_modules/**"
--exclude="**/node_modules/"
etc etc.

None work!

Problem only happens with sync, when running copy --recursive true the exclude string is honoured and works

Which version of the AzCopy was used?

azcopy version 10.1.0

Which platform are you using? (ex: Windows, Mac, Linux)

Windows 10

What command did you run?

azcopy sync nodejs-demoapp https://bcmisc.blob.core.windows.net/backup/dev1\?sv=REMOVED --exclude="*node_modules*"

What problem was encountered?

Exclude doesn't work

How can we reproduce the problem in the simplest way?

Create a directory with a subdirectory inside it (and some test files inside that), and try to sync the main directory while exluding the sub-directory

Have you found a mitigation/solution?

No

[zezha-msft]

Hi @benc-uk, thanks for reaching out! And sorry for the inconvenience.

The difference in sync's behavior (compared to cp) was very much intentional for the sake of simplifying the user experience. Both ./azcopy sync --help and the wiki explain the meaning of the exclude flag:

exclude files whose name matches the pattern list. Example: .jpg;.pdf;exactName

It's good to hear your feedback though. We can perhaps rename the current flag to --exclude-files, and add a new flag --exclude-directories to address your needs. What do you think?

[zezha-msft]

Or alternatively, we could do --exclude-pattern which is a list of patterns that match to the file names, and --exclude-path which is a list of literal paths.

The reason why I don't think we should mix up the two lists, is because it might be confusing as to what's the expected behavior. Ex: if the user is working on MacOS and wishes to exclude all .DS_Store files, and if we only have 1 flag, would all .DS_Store files be ignored, or just the one under the given path?

[benc-uk]

Hi @zezha-msft, ok this makes some sense now.
I think the --exclude-directories flag would be a great option to add

[svanmoerkerke]

any update on this?

[JohnRusk]

@zezha-msft re whether we have any update at this stage.

[zezha-msft]

Thanks for following up @svanmoerkerke!

This item is being tacked in our backlog, and we'll try to schedule it as soon as possible.

[rugheid]

We would also really like more extensive include and exclude support. Our preference goes to patterns that match the entire file path (relative to the sync location). In blobxfer for example, the include and exclude patterns implement general glob patterns, which are much less limiting.

[JohnRusk]

A version of this (with literal paths but no globbing) is planned for release 10.3

[zezha-msft]

This has been fixed in 10.3.0.

[coding-lee]

Hello @zezha-msft ,
The issue seems still not fix. We can use the 'exclude-path' flag only in cp command rather than sync command.
My Azcopy version is v10.3.1. Could you help me to check it? Thx.

@benc-uk

[zezha-msft]

@coding-lee sorry! Thanks for letting us know. It appears that we did not expose this flag for sync correctly.

[zezha-msft]

@coding-lee I've added the appropriate work item to keep track of this for the next release 10.4.0.

[coding-lee]

@zezha-msft Thank you for your kindly help.

[nerijusk]

when is 10.4 release planned? this --exclude-path option would really be useful in sync command.

[JohnRusk]

Exclude-path for sync is now in 10.3.3. We don't publish dates, sorry.

[nerijusk]

Exclude-path for sync is now in 10.3.3. We don't publish dates, sorry.

@JohnRusk didn't quite get your comment. is 10.3.3 released? i could not find it in the release page. or did you mean it is planned to be released sometime soon with exclude/include path feature?

[JohnRusk]

Sorry. I wasn't clear. 10.3.3 will be released sometime soon.

[pniederlag]

just ran into this as well, great to see it will be fixed soon

[JohnRusk]

10.3.3. was released today, and includes this feature for sync.