Azure-storage-azcopy: Transfers not retried on 'unexpected EOF'

Created on 11 Jun 2019 · 14Comments · Source: Azure/azure-storage-azcopy

Which version of the AzCopy was used?

10.1.2

Which platform are you using? (ex: Windows, Mac, Linux)

Mac

What command did you run?

azcopy cp --recursive https://myaccount.blob.core.windows.net/mycontainer ./

What problem was encountered?

There were a few dozen failed transfers out of a few thousand that were not retried. They all failed with "unexpected EOF".

How can we reproduce the problem in the simplest way?

I am not aware of an easy way to reproduce this.

Have you found a mitigation/solution?

We can get the missing files with azcopy sync.

It looks like azcopy only considers net.Error to be retryable. It would be nice if io.ErrUnexpectedEOF was retryable.

Source

juve

Most helpful comment

Hi @sjuratov @AtillaTv8, this issue was fixed in #859, and will be released shortly in 10.5.

zezha-msft on 7 Jul 2020

👍2

All 14 comments

Hi, @juve. AzCopy failing to transfer to a relative directory destination is a currently known issue and is solved in PR #422. We're currently working on getting it into our next release.

adreed-msft on 11 Jun 2019

@adreed-msft I don't actually know if the destination in our case was relative or not. Most of the transfers were successful, so it wasn't a complete failure. The error was 'unexpected EOF' in the log, which doesn't seem related to the relative path issue.

juve on 11 Jun 2019

Oh? Interesting. I'll take a look here in a bit.

adreed-msft on 11 Jun 2019

@adreed-msft Any update on this? Is it logged in our backlog for investigation?

JohnRusk on 6 Aug 2019

BTW, I've been corresponding, via MSFT internal channels, with staff supporting a customer where this has happened. Seems to be something in the network path (e.g. a proxy) truncating _something_. Still in dialogue to try to figure out what is being truncated (there are two or three identified possibilities).

JohnRusk on 19 Sep 2019

👍1

The only thing I found in the other support case was that it was almost certainly caused by a proxy server.

@juve Are you still seeing this error with more recent versions of AzCopy?

JohnRusk on 15 Nov 2019

I am getting this error with
azcopy version 10.3.2

2019/11/22 02:16:27 INFO: [P#11-T#5479] Error reading body of reply. Next try (if any) will be Try=2. Will NOT retry. Error: unexpected EOF.

guri-s on 25 Nov 2019

We've recently found that there's a bug, in which AzCopy should retry the operation, but for this specific type of failure, it does not. I don't have a release date for any fix yet, but if you choose "watch" on this page https://github.com/Azure/azure-storage-azcopy/releases you'll be notified of all releases.

JohnRusk on 25 Nov 2019

👀1

Had to move this out of release 10.3.3. and into 10.4.0, because the necessary change is bigger than expected, including changes to the blob and Azure Files SDKs in addition to AzCopy itself.

Sorry we can't get it into 10.3.3.

JohnRusk on 10 Dec 2019

Has this issue been resolved? I don't see it documented @ https://github.com/Azure/azure-storage-azcopy/releases

I am trying to copy big file (100GB) from Azure blob storage account to VM in Azure. There is a proxy in between. Copy starts but then it errors out after some time (always less then 20% completed). In log I can see following:

**2020/07/01 06:50:43 INFO: [P#0-T#0] Error reading body of reply. Next try (if any) will be Try=2. Will NOT retry. Error: unexpected EOF. Offset: 20792950784 Count: 2408448 URL: https:// 06:50:43 INFO: [P#0-T#0] Error reading body of reply. Next try (if any) will be Try=2. Will NOT retry. Error: context canceled. Offset: 20112736256 Count: 3145728 URL: https:// 06:50:43 ERR: [P#0-T#0] DOWNLOADFAILED: https:// : 000 : unexpected EOF. When Enqueuing chunk. X-Ms-Request-Id:

Dst: %5C%5C?\E:\temp\**

Command I am executing:
azcopy copy https:// e:\temp

Azcopy version is azcopy_windows_amd64_10.4.3

sjuratov on 1 Jul 2020

Has this issue been resolved? I don't see it documented @ https://github.com/Azure/azure-storage-azcopy/releases

I am trying to copy big file (100GB) from Azure blob storage account to VM in Azure. There is a proxy in between. Copy starts but then it errors out after some time (always less then 20% completed). In log I can see following:

**2020/07/01 06:50:43 INFO: [P#0-T#0] Error reading body of reply. Next try (if any) will be Try=2. Will NOT retry. Error: unexpected EOF. Offset: 20792950784 Count: 2408448 URL: https:// 06:50:43 INFO: [P#0-T#0] Error reading body of reply. Next try (if any) will be Try=2. Will NOT retry. Error: context canceled. Offset: 20112736256 Count: 3145728 URL: https:// 06:50:43 ERR: [P#0-T#0] DOWNLOADFAILED: https:// : 000 : unexpected EOF. When Enqueuing chunk. X-Ms-Request-Id:

Dst: %5C%5C?\E:\temp**

Command I am executing:
azcopy copy https:// e:\temp

Azcopy version is azcopy_windows_amd64_10.4.3

hello sjuratov, did you solve the problem? I encountered the same problem.
I cannot download more than 20% for large files. Installed Storage explorer latest version 1.14.0 problem not solved.
Best regards