Azure-storage-azcopy: [Question] Error: Combination 'Localfile' not supported for sync command (File Storage)

[Kapanther]

Azcopy 10.0.2 Preview - Win 7 - Powershell

The following string works for a "copy" but when using sync it throws the error further below
.\azcopy sync "C:\GCDS_dev" "https://azgcdsdevst1.file.core.windows.net/gcdsbuild/GQtest?<retracted_key>" --recursive

error parsing the input given by the user. Failed with error source 'C:\GCDS_dev' / destination 'https://azgcdsdevst1.fi
le.core.windows.net/gcdsbuild/GQtest?"retracted_key"' combination 'LocalFile' not supported for sync command

Any Ideas?

[prjain-msft]

@Kapanther Currently we don't support sync between Local and Azure File. We support sync only between Local and Azure Blob. But we plan to implement sync between Local and Azure File eventually.

[prjain-msft]

No Support for sync between Local and Azure Files.

[Kapanther]

@prjain-msft thanks for confirming. Tested against blob storage and it works fine. (a bit slow, but possibly my awful australian internet)

[mazong1123]

Do we have ETA for adding support of syncing between Local and Azure Files?

[oscarwest]

I've been waiting on this for a couple of months now, looking forward to it being implemented.

[zezha-msft]

Hi @oscarwest, I apologize for the inconvenience.

We had to do some major refactoring, so this feature is not implemented yet. But it will much easier to implement once we get to it.

[Kapanther]

@zezha-msft nicest guy ever... I hope they pay you well

[oscarwest]

@zezha-msft Sounds good :)

[oscarwest]

Is this still planned? @zezha-msft

[JohnRusk]

It's on our approved backlog of work, but we don't have any planned date to share yet. Is your need for it time sensitive, with a particular deadline? If so please let us know here or via email. Can't promise anything of course, but all information is helpful in planning.

[oscarwest]

We run some stuff in AKS which mounts from Fileshares, in our CD pipe we'd love to have this sync capability so we never get diffs. It's not necessarily time sensitive but would make our life a lot easier :)

[oscarwest]

If we could integrate this with Azure DevOps it would be even greater, right now we're deploying from Octopus Deploy with the binary

[JohnRusk]

Any suggestions on what such integration, with DevOps, would look like?

[oscarwest]

Maybe expanding on something like this?
https://github.com/microsoft/azure-pipelines-tasks/blob/master/Tasks/AzureFileCopyV1/README.md

[JohnRusk]

So that one is using AzCopy v8, right? And you'd be looking for:

1. The same thing, but with v10... AND
2. With "sync" supported, from Local to Azure Files?

Is that correct? If so, would you mind popping over to the azure-pipelines-tasks GitHub project, and logging an issue there to request v10 support? You can include a link back to this issue, with a comment that its Local to Files sync that you most want.

[oscarwest]

Yes that's right, sure i'll create an issue over there so you guys can _sync_

[JohnRusk]

:-)

[Symbianx]

We also need this feature. Is there a workaround to have the same behavior with the azcopy copy command?

[JohnRusk]

@Symbianx The copy command already works from local disk to Azure Files. Is that what you were asking about, or are you looking for something else?

[HainanZhao]

We also need this feature. Is there a workaround to have the same behavior with the azcopy copy command?

A temporary solution I used is mount Azure File to a local machine, and use robotcopy to sync(To be exact, it's mirroring).
Robocopy S:\FolderPath\ T:\FolderPath\ /MIR /FFT /Z /XA:H /W:5 /MT:8

[Symbianx]

@JohnRusk From what I understood, the azcopy copy doesn't remove the files that no longer exist (like rsync --delete).

Our use case is pushing application backups to the Azure File Share from a container running in kubernetes, we also need to remove the files that don't exist locally anymore (hence the need to sync).

In one of our environments we are mounting the file share inside the container and we rsync the directories, however this forces us to run the container in privileged mode which is not ideal but works.
On the other environment the only way to access the internet is through an http_proxy, but unfortunately we can't mount the file system through the proxy.

[JohnRusk]

@Symbianx. Yes, it does sound like local->Files sync would be ideal for you there. Sorry its not ready yet!

Maybe you could to a workaround where the copying is done by AzCopy but the deletion is done differently. How are you authenticating? With a SAS url/token? If so, it's _probably_ possible (with a little work) to call the rest API directly from script, to list the contents of the file share. Then, still in script, you could compare the local file listing with the one retrieved via the rest API, and then delete the ones that need to be deleted one by one (probably again with the REST API).

In theory you could do the same with other authentication types, but they are not as easy to script as SAS.

That's the best workaround I can think of right now. (And I'm not sure its terribly good... but its the best I can think of at the moment).

Note that you can't compare last modified dates with copy. You either re-copy everything, or else sett overwrite=false and don't copy anything that already exists (last modified dates are not checked, only existence is checked).

[Symbianx]

@JohnRusk Thanks for the suggestion, we are using SAS.
For now, we ended up cleaning the entire directory with azcopy remove --recursive and uploading all of the files over again.
This is not ideal but since the files are relatively small I think this is a nice compromise until sync supports Local->Files.

Do you think there will be a downside that we may not be aware of?

[JohnRusk]

That sounds fine.

[coopers98]

Just tossing in a vote of support for this functionality!

[northcide]

Parity between robocopy and azcopy would make cloud migrations much simpler. +1 for local to file sync.

[dhaschur]

Since Azure Files Fileshare with On premises AD ACL support has gone GA, I'm guessing a lot more will start missing this feature :) Big fileshares need the incremental sync otherwise we need a few days of down time for the share, which is never going to happen.

[timja]

Any progress on prioritisation for this?

[albertopiva-tv]

This is a must have functionality. We have to sync GBs of small files of which only a few change every week. Now we have to copy everything every time.

[JohnRusk]

@rpohane

[EdiFirst]

Hey Guys, its can help me alot.Is there some update about azcopy sync with Azure FileShare ?

[JohnRusk]

cc @amishra-dev

[mili988]

Hi Guys, do we have any new status here?

[amishra-dev]

We will do this for out 10.9 release. @adreed-msft to make it happen.

[wsandmanw]

Amishra-dev;

Any chance of getting a hight priority assigned to this? This feature has been in request since 2018. Getting bi-direction sync is a featuer much-needed in this feature set.

[zezha-msft]

Hi @wsandmanw, we've already committed to a scope for 10.8, so 10.9 is where we are adding the high priority items such as this feature.

[reji123]

Thanks for the update. Any update on when 10.9 will be released ?

[kman2123]

+1

[kman2123]

@zezha-msft any word?

[zezha-msft]

Thanks for following up, we shuffled the numbers a bit and the original 10.9 is now 10.10. We are targeting to land it in early Q2.

[reji123]

ok...thanks for the update...

[mohsha-msft]

Hey @Kapanther,

We've addressed this issue in the recent release of AzCopy 10.10.0. Please use the same and do reach out to us for more information.