Azure-storage-azcopy: syncing large containers misses some files

[nofunatall]

azcopy v10.3.1

azcopy sync --delete-destination true "https://${SRC_ACC}.blob.core.windows.net/${SRC_CONT}${SRC_SAS}" "https://${DST_ACC}.blob.core.windows.net/${DST_CONT}${DST_SAS}"

It seems the sync command misses some files when running on very large containers.
In this case I was using the --delete-destination flag and I have not tested to see if the problem exists without using delete.

So at this point I am not sure if delete is deleting items it shouldn't or if the copy portion of syncing is actually missing files.

[JohnRusk]

Ohh, that shouldn't happen. Looks like you're using the default logging level, so there might be some useful information in the log. But if the container is really large, it may become hard to process the log. Roughly how many files are we talking about?

[nofunatall]

I've yet to enumerate the entire container.
It's easily millions, maybe billions of files.

[zezha-msft]

Hi @nofunatall, could you please double check whether the missing files were created after the job started running? If they weren't present when scanning first started, there's no guarantee that the service will let us know about their existence.

The other suggestion would be to run the sync job again, and see if the missing files are transferred this time.

[nofunatall]

Just a note:
I've run a copy over night there are still missing files.

Further explanation:
The source account is a partially active account attached to a dev environment.
The destination account is completely inactive and attached to my own sandbox environment.

I am checking for missing files by navigating my sandbox environment (destination of copy) and seeing files missing that definitely exist on the source account.

These files are all tracked within a few select tables within MariaDB and my sandbox is using a copy of that DB that is some weeks older then the dev environment acting as the source.

Without going into details what this essentially means is it doesn't matter when files were created on the dev source environment - As long as I do the azcopy operation after restoring a database from the source and not before there should be no missing files showing on the sandbox (destination) environment I am using.

[JohnRusk]

Thanks for the detailed description. One question, how does that database restore influence things, given that sync only looks at blobs, their existence, and their creation times?

[nofunatall]

It was more answering @zezha-msft about files being present or not when starting a scan.

There are always more files in Azure then there are records on environment DB's as files upload first before the record is added to the DB. (Files are never deleted ever either)

In short there is no other logical explanation why the destination environment is missing files other then the azcopy sync/copy missing items.

[zezha-msft]

Hi @nofunatall, thanks for the additional context.

Could you please provide examples of missing files? Do you see patterns in them? Have you checked independently that they indeed existed at the time of scanning? (The DB is a secondary source of truth, the primary source of truth is still the Blob service itself)

[nofunatall]

Seems completely random for which files are missed - possibly newer files are missed more often then older ones but it's very time consuming to manually check this.

I can say the missing blobs definitely exist in the source account at the time of the sync/copy.

[landro]

Could this issue actually be due to a bug in either

• the blob service API not returning all objects or
• the go binding not picking up all of them?

We're about to both copy and sync millions of files, and we really have to make sure this scenario is well tested by you guys.

[JohnRusk]

@landro I'm attempting to replicate @nofunatall's issue.

[zezha-msft]

Hi @nofunatall, the DB is still a secondary source of truth though, could you please verify the blob endpoint directly? Maybe through Storage Explorer?

[JohnRusk]

@nofunatall Will you still have the log file from the AzCopy job? If so, you could see whether this is a delete issue (or a copying issue) by searching the log for deletions:

In Powershell:

select-string DELETE <pathToLogFile> | Out-File <pathToDesiredNewOutputFile>

On Linux the command is similar, but with grep.

FYI, I have not yet managed to reproduce your issue, but I'll continue trying.

[JohnRusk]

It would also pay to check the summary at the end of the log file, to make sure that AzCopy did not report any errors:

get-content <pathToLogFile> -tail 20 

[nofunatall]

Running another sync today to grab some logs, as I can't remember what the last log file was.

[JohnRusk]

Thanks. I really appreciate you doing that. BTW, I've spent a whole day trying to reproduce the issue - i.e. missing files in sync - and have failed to do so.

The key things to look for in the log are probably to start with the name of a missing file, i.e. one that should have been synced but wasn't, and then see if that filename appears in the log. E.g. did AzCopy move it then delete it? Did AzCopy never move it at all?

[nofunatall]

I've run the sync, but can't spot missing files now - this could be because I've now run a sync/copy several times now.

I will delete the dataset tomorrow, and run a sync just once as it will be easier to spot missing files if any truly are missed.

[JohnRusk]

Any update on this @nofunatall ?

[nofunatall]

I can't seem to repro the issue now.
Perhaps server load or some other external factor was the cause.

Go ahead and close the issue.

[JohnRusk]

Thanks for retesting it @nofunatall . Given your comments above, and the fact that nothing similar has ever been seen in our own testing or reported by anyone else, we'll conclude that the cause of the initial anomaly was not an AzCopy bug.

[JohnRusk]

@landro There does not appear to be any problem in this issue.

FYI, our own testing within the team has used a range of payload sizes including jobs with many millions of files. We've never seen any files get missed but there a couple of known performance issues once you get up to 10's of millions of files in a single AzCopy job. So if you plan to move 100's of millions of files, it's wise to partition the work into serveral different AzCopy jobs instead of doing it one big job. If breaking the work up is not practical for you, please contact me. My email address is on my profile page.