Azure-storage-azcopy: AzCopy on ARM/Synology crashes

[silaspepperbyte]

Which version of the AzCopy was used?

AzCopy V10.1.2

Which platform are you using? (ex: Windows, Mac, Linux)

Linux ARM 32 bits. Synology RS816

What command did you run?

sync /volume1/veeam-iscsi https://*.blob.core.windows.net/?* --recursive=true --delete-destination=true

What problem was encountered?

Crash after starting the job

How can we reproduce the problem in the simplest way?

Compile for ARM and run on a Synology machine. Logs attached.
SynologyLog.txt
AzcopyLog.txt

Have you found a mitigation/solution?

No, AzCopy wont continue

[ziXet]

Would you please upload the azcopy file that you built for ARM? I wanted to run the azcopy on our ReadyNAS but realized that only x64 arch is available for download.

[JohnRusk]

@silaspepperbyte We recently found (and fixed) and issue related to running on ARM. The fix is to use the very latest jiacfan/keyctl release. See comments for https://github.com/Azure/azure-storage-azcopy/issues/415 for details. Were you using the latest jiacfan/keyctl?

@ziXet We (Microsoft) do not currently distribute an ARM build. However its very easy to make your own, by cross compiling from a normal Window or Linux machine. If you need more info on how to do that please reply here and we'll add some details to this thread (or somwhere).

[silaspepperbyte]

@silaspepperbyte We recently found (and fixed) and issue related to running on ARM. The fix is to use the very latest jiacfan/keyctl release. See comments for #415 for details. Were you using the latest jiacfan/keyctl?

@ziXet We (Microsoft) do not currently distribute an ARM build. However its very easy to make your own, by cross compiling from a normal Window or Linux machine. If you need more info on how to do that please reply here and we'll add some details to this thread (or somwhere).

Dear JohnRusk,

The version I compiled already includes this update to keyctl.

@ziXet here is the AzCopy for ARMv7
azure-storage-azcopy.zip

[Edit from maintainer zezha-msft: to clarify, this is NOT an official build from Microsoft, please exercise caution]

[ziXet]

Thanks @silaspepperbyte
I have tried it on a different NAS Linux machine. it works fine for small containers. But failed for a container that has around 16K blobs.
Here is the log: https://pastebin.com/w0fkY1b6

[ziXet]

@silaspepperbyte We recently found (and fixed) and issue related to running on ARM. The fix is to use the very latest jiacfan/keyctl release. See comments for #415 for details. Were you using the latest jiacfan/keyctl?

@ziXet We (Microsoft) do not currently distribute an ARM build. However its very easy to make your own, by cross compiling from a normal Window or Linux machine. If you need more info on how to do that please reply here and we'll add some details to this thread (or somwhere).

I think having some guides for cross-compiling the tool would be really useful!

[JohnRusk]

@ziXet thanks to the log. The issue appears to be with a concurrently locking construct. I'm guessing its happening only on larger containers because there is more concurrency (more parallel operations) there. But that's just a guess.

I have added two items to our backlog of work: one to investigate the error in your log and one to add some docs sometime about how to cross compile. Right now, I don't have any ETA for when we'll be able to work on those two things.

[Team, please leave this issue open until BOTH are resolved - crash and docs]

[silaspepperbyte]

@ziXet thanks to the log. The issue appears to be with a concurrently locking construct. I'm guessing its happening only on larger containers because there is more concurrency (more parallel operations) there. But that's just a guess.

I have added two items to our backlog of work: one to investigate the error in your log and one to add some docs sometime about how to cross compile. Right now, I don't have any ETA for when we'll be able to work on those two things.

[Team, please leave this issue open until BOTH are resolved - crash and docs]

I would agree that large files would cause this. My files are all large files which would explain the crash.

[JohnRusk]

See also: https://github.com/Azure/azure-storage-azcopy/issues/882