Azure-storage-azcopy: Unable to download complete set of files due to "412 The condition specified using HTTP conditional header(s) is not met"

[ayeltsov]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

10.4.3

Which platform are you using? (ex: Windows, Mac, Linux)

Windows

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

s:\temp\azcopy.exe copy "https://accountname.blob.core.windows.net/prefetch/data/*?se=2020-06-01T20%3A54%3A01Z&sp=rl&sv=2018-03-28&sr=c&sig=<...>" "Y:\PrefetchData" --overwrite=prompt --recursive --from-to=BlobLocal --preserve-last-modified-time

What problem was encountered?

Not able to download 130K files - every time some files (80 to 170) get errors.

Below are 3 attempts, note that every time some percentage fails.
It is different files every time.
I don't see a retry option for this situation.

1)

99.6 %, 131189 Done, 130 Failed, 828 Pending, 0 Skipped, 132147 Total, 2-sec Throughput (Mb/s): 21.6107

Job 6c0052c9-9788-1740-6c44-d554cf861184 summary
Elapsed Time (Minutes): 4.0987
Number of File Transfers: 132147
Number of Folder Property Transfers: 0
Total Number of Transfers: 132147
Number of Transfers Completed: 132016
Number of Transfers Failed: 131
Number of Transfers Skipped: 0
TotalBytesTransferred: 610341232
Final Job Status: CompletedWithErrors

2)

100.0 %, 131982 Done, 167 Failed, 1 Pending, 0 Skipped, 132150 Total,

Job f3f892fc-03db-e041-7a8c-3e062e7d8cfb summary
Elapsed Time (Minutes): 8.2499
Number of File Transfers: 132150
Number of Folder Property Transfers: 0
Total Number of Transfers: 132150
Number of Transfers Completed: 131983
Number of Transfers Failed: 167
Number of Transfers Skipped: 0
TotalBytesTransferred: 609848351
Final Job Status: CompletedWithErrors

3)

100.0 %, 132067 Done, 83 Failed, 1 Pending, 0 Skipped, 132151 Total,

Job 396c74ad-69d5-ed4a-56ff-f413c3006e28 summary
Elapsed Time (Minutes): 2.1031
Number of File Transfers: 132151
Number of Folder Property Transfers: 0
Total Number of Transfers: 132151
Number of Transfers Completed: 132068
Number of Transfers Failed: 83
Number of Transfers Skipped: 0
TotalBytesTransferred: 609147498
Final Job Status: CompletedWithErrors

Log:

2020/05/02 21:55:50 AzcopyVersion 10.4.3
2020/05/02 21:55:50 OS-Environment windows
2020/05/02 21:55:50 OS-Architecture amd64
2020/05/02 21:55:50 Log times are in UTC. Local time is 2 May 2020 14:55:50
2020/05/02 21:55:50 Job-Command copy https://accountname.blob.core.windows.net/prefetch/data/*?se=2020-06-01t20%3A54%3A01z&sig=-REDACTED-&sp=rl&sr=c&sv=2018-03-28 Y:\PrefetchData --overwrite=prompt --recursive --from-to=BlobLocal --preserve-last-modified-time
2020/05/02 21:55:50 Number of CPUs: 8
2020/05/02 21:55:50 Max file buffer RAM 4.000 GB
2020/05/02 21:55:50 Max concurrent network operations: 128 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2020/05/02 21:55:50 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2020/05/02 21:55:50 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2020/05/02 21:55:50 Max enumeration routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2020/05/02 21:55:50 Max open files when downloading: 2147483066 (auto-computed)
2020/05/02 21:55:50 Any empty folders will not be processed, because source and/or destination doesn't have full folder support
2020/05/02 21:55:50 JobID=6c0052c9-9788-1740-6c44-d554cf861184, credential type: Anonymous
2020/05/02 21:55:50 scheduling JobID=6c0052c9-9788-1740-6c44-d554cf861184, Part#=0, Transfer#=0, priority=0
<...>
2020/05/02 21:56:01 ERR: [P#0-T#5359] DOWNLOADFAILED: https://accountname.blob.core.windows.net/prefetch/data/filename.v3.txt?se=2020-06-01t20%3A54%3A01z&sig=-REDACTED-&sp=rl&sr=c&sv=2018-03-28 : 412 : 412 The condition specified using HTTP conditional header(s) is not met.. When Downloading response body. X-Ms-Request-Id: b0413918-801e-003d-50cc-20cd4e000000

Dst: %5C%5C?\Y:\PrefetchData\filename.v3.txt

How can we reproduce the problem in the simplest way?

Not sure. We have 5000+ machines modifying these files so you'd have to set up some sort of multithreaded writer that modifies files.

Have you found a mitigation/solution?

Not with azcopy.

[JohnRusk]

@zezha-msft My guess is that this is the download equivalent of the issue we discussed last week, with uploads files modified after scan time. Perhaps we can apply a similar solution here.

[ayeltsov]

files modified after scan time

Most definitely (notice that tidbit about 5000 machines all updating various files in the folder)

[jibonilla03]

Having same problem with $ sudo azcopy --version
azcopy version 10.11.0