Azure-storage-azcopy: Why "INFO: Scanning" takes too long? How to speed it up?

[ldepaiva]

Which version of the AzCopy was used?

10.50

Note: The version is visible when running AzCopy without any argument

Which platform are you using? (ex: Windows, Mac, Linux)

Linux / Mac

What command did you run?

./azcopy cp 'from-blob-string' 'to-blob-string' --recursive --include-after 2020-07-14

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

What problem was encountered?

The whole storage has 6kk file but I'm trying to migrate just the ones from 2020-07-14 which is ~12k files. The issue is that it takes about 3h at the INFO: Scanning.... Does it have to scan all the 6kk files in order to copy those ~12k files? Is there any argument that I can use to speed things up? How to sync two blob storages?

I have already migrated those 6kk files to another blob storage and it took 15h (not sure if it includes the scanning time) but I need to migrate the new remaining files in order to move over to the new blob storage.

How can we reproduce the problem in the simplest way?

Just try to migrate some portion of 1kk files from blob storage A to blob storage B

Have you found a mitigation/solution?

No

[zezha-msft]

Hi @ldepaiva, thanks for reaching out.

To confirm, what do you mean by 6kk? Do you have 6 million blobs?

Do you have any hierarchy in the container? Are all the files at the root directory?

[ldepaiva]

Hey, @zezha-msft. Yes, they are 6 million blobs and the containers are all at the root level. There are probably 1 million containers at the root level and each container has a handful blob.

[seanalexander]

IS there an option to not scan, and just copy the files provided in a list?