Azure-storage-azcopy: Copy fails (CannotVerifyCopySource) when storage account firewall is enabled

[nfriday]

Using azcopy 10.3.2 (Windows), with a storage account firewall enabled on the source and destination, copy (or sync) between file shares fails with CannotVerifyCopySource.

azcopy copy https://nicktestsource.file.core.windows.net/source/test.txt?sv=2018-11-09&sig=<REDACTED>=2019-11-21T10%3A07%3A46Z&se=2020-11-21T10%3A07%3A46Z&srt=sco&ss=f&sp=racupwdl https://nicktestdest.file.core.windows.net/dest/test.txt?sv=2018-11-09&sig=<REDACTED>&st=2019-11-21T10%3A01%3A58Z&se=2020-11-21T10%3A01%3A58Z&srt=sco&ss=f&sp=racupwdl

Copy fails with:

INFO: Authentication failed, it is either not correct, or expired, or does not have the correct permission -> github.com/Azure/azure-storage-file-go/azfile.newStorageError, /home/vsts/go/pkg/mod/github.com/!azure/[email protected]/azfile/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=CannotVerifyCopySource) =====
Description=This request is not authorized to perform this operation.
RequestId:b50a48eb-701a-0026-2f53-a0d39d000000
Time:2019-11-21T10:07:49.6641367Z, Details: 
   Code: CannotVerifyCopySource
   PUT https://nicktestdest.file.core.windows.net/dest/test.txt?comp=range&se=2020-11-21t10%3A07%3A46z&sig=-REDACTED-&sp=racupwdl&srt=sco&ss=f&st=2019-11-21t10%3A07%3A46z&sv=2018-11-09&timeout=901
   Content-Length: [0]
   User-Agent: [AzCopy/10.3.2 Azure-Storage/0.6.0 (go1.13; Windows_NT)]
   X-Ms-Client-Request-Id: [3f5dbe62-5b0a-4528-6a02-3b891c5f387b]
   X-Ms-Copy-Source: [https://nicktestsource.file.core.windows.net/source/test.txt?se=2020-11-21t10%3A07%3A46z&sig=-REDACTED-&sp=racupwdl&srt=sco&ss=f&st=2019-11-21t10%3A07%3A46z&sv=2018-11-09]
   X-Ms-Range: [bytes=0-4]
   X-Ms-Source-Range: [bytes=0-4]
   X-Ms-Version: [2019-02-02]
   X-Ms-Write: [update]
   --------------------------------------------------------------------------------
   RESPONSE Status: 403 This request is not authorized to perform this operation.
   Content-Length: [248]
   Content-Type: [application/xml]
   Date: [Thu, 21 Nov 2019 10:07:48 GMT]
   Server: [Windows-Azure-File/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [3f5dbe62-5b0a-4528-6a02-3b891c5f387b]
   X-Ms-Error-Code: [CannotVerifyCopySource]
   X-Ms-Request-Id: [b50a48eb-701a-0026-2f53-a0d39d000000]
   X-Ms-Version: [2019-02-02]

Problem can be replicated by attempting a copy between shares on different storage accounts which both have a storage account firewall configured.

Disabling the storage account firewall on both the source and destination works.

[adreed-msft]

Hi there! Because we make use of the Put*FromURL APIs available in Azure storage, the destination should be capable of reaching the source storage account.

Please ensure this is possible, and try again.

[nfriday]

The source and destination are both Azure files shares.

It is not possible (as far as I know) to configure the source storage account's firewall to allow access from the destination's storage account.

Am I therefore correct to understand that what I am trying to do is impossible?

[JohnRusk]

It's only impossible in the short term. A change is currently rolling out to the Azure Files service. Once that change is rolled out, it should work like this:

1. The machine where AzCopy is running needs to be able to access the source account. (Naturally).
2. The machine running AzCopy needs to be able to access the destination account (Naturally).
3. As long as the above conditions are met, then the account to account copy will work (with no other firewall config). Effectively it will use the IP address of the AzCopy client machine to authorize the source to destination traffic. I.e. it will magically and securely work, without you needing to tell one storage account about the other.

I don't know exactly when this change will finish rolling out. If you have a particularly urgent need, please email me and I'll make some inquiries for you. My email address is in my profile page.

cc @adreed-msft This config change is good news, and will resolve the issue you mentioned above with no effort from users.

[JohnRusk]

The config change is not yet available. I'm investigating other workarounds. Please reply here if you need details.

[nfriday]

Hi John, I'm interested in any workarounds you might find!

[JohnRusk]

I'm making some enquiries to find a workaround using AzCopy v10, if possible. In the meantime, you could consider temporarily using the older version (AzCopy version 8.1). It has totally different command line syntax, and its Account to Account copy operations are not as quick and convenient as V10s because it uses older Storage APIs. But, conveniently in this case, the older APIs are not affected by this issue. So I'm told they should work right now in the above-mentioned Storage Firewall configuration.

By default, V8 asks the Storage Service to copy asynchronously from the source to destination. There's no SLA for how long the copy can take. IIRC, it can take up to a week (but is typically much less. Details of the Async copy are documented on the Azcopy v8 documentation page). Or, if you use the /sync option, it downloads the files to the machine running AzCopy and streams them immediately back up to the destination. (If using /sync, it’s probably best to run AzCopy on a medium-sized cloud VM, rather than on-premise, to avoid any potential bandwidth restrictions of your on-prem network link).

Note that V8 is effectively a totally different product from v10, with a different command line syntax.

[JohnRusk]

Update: there isn't any short-term workaround in V10. I'm re-opening this issue, to remind us to post an update here when the Service-side issue is resolved. When that is resolved, v10 will work and we can close this issue.

[amarnathkarthik]

@JohnRusk is there a workaround available to copy the data between storage accounts with selected network enabled?

[JohnRusk]

For users who are using Storage Firewall _without_ Private Link, I believe the server-side resolution should be enabled in most (if not all) places by about now. But, if you also have Private Link, then there is still an issue. See https://github.com/Azure/azure-storage-azcopy/issues/1003.

[aplex]

@JohnRusk , can AzCopy V8 work when there is a firewall and a private link, or it has the same problems as AzCopy V10?

[JohnRusk]

I believe issue #1003 applies to AzCopy v8 too, since it uses one of the affected APIs, but I have not tested. If you have followup questions, please ask them on that issues thread. Thanks.

[zffocussss]

azcopy version 10.3.4 can work ,while the new version v10.8.0 can not work @JohnRusk

[zffocussss]

it is strange linux v10.8.0 can work as well.but it failed some hours ago.what happened?

[JohnRusk]

@zffocussss please log a new issue for your recent failure.

[bahadir-be]

I have a similar issue.

[zezha-msft]

Should be fixed by now: https://azure.microsoft.com/en-us/updates/copy-blob-support-over-private-endpoints/