Azure-pipelines-tasks: AzureFileCopy support AzCopy (v10) sync command

[eelkram]

The AzureFileCopy task should be updated to support the new AzCopy v10 tool and the new sync command. This would be extremely useful in conjunction with the static website hosting on Azure Storage capabilities.

[rajatagrawal-dev]

@eelkram Can you give more details on your scenario? Where do you see the need for the sync command and how will it complement the static website support?

[eelkram]

@rajatagrawal-dev - I use AzCopy to deploy a React front end on Azure Storage that is served up via the static website support and fronted via an Azure CDN. During the build process the artifacts are dynamically named with a hash to eliminate caching issues. When using the current AzCopy commands it moves the new files up to Azure Storage but never removes the old files. I would like to be able to use the sync command to copy out the new files, update existing files and remove the ones that no longer exist without having to write custom code.

Let me know if you have any other questions.

[rajatagrawal-dev]

@eelkram Can you work around for now by adding a task before Azure File Copy task? That task can be an Azure CLI task or Azure Powershell task. You can write a one line command in here to delete the container.

[mark-studer]

Yep, I can work around it for now. Once this is added I will switch to using the sync command.

[albernhagen]

Is there any timeline as to when this will be added? I'd prefer to not delete the storage container before deploy if I can help it

[oscarwest]

We would benefit greatly from this as well.
Ref to issue at their repo:
https://github.com/Azure/azure-storage-azcopy/issues/75

[rajatagrawal-dev]

After going through the documentation of AzCopy v10, I think there are a lot of changes we might have to do in the task in order to upgrade to that. It might change the current task design in order for us to support the new features of v10.

This is not something that we have planned for as of now. Hence, I can't comment on any timelines here.
I will update this issue once we have any plans for this. Sorry for the inconvenience.

[MaxMelcher]

Would you accept a PR for this @rajatagrawal-dev?

[rajatagrawal-dev]

@MaxMelcher We won't update the current task (Azure File Copy v3) to support AzCopy v10 as it will most likely break existing users. For AzCopy v10 support, we will have to create a new major version of the task (Azure File Copy v4). Since it is a major version, the task will have to be developed internally.

But you are free to create a new task which supports AzCopy v10 and upload to your own Azure DevOps instance. Use tfx-cli for that purpose.

[rwerlang]

I'm having the same issue publishing a reactjs to a storage account.
The problem to workaround is that PowerShell does not support using a container named as $web, because $ is treated as invalid.

Any other workarounds to delete the blob before using the AzureFileCopy?

[MaxMelcher]

You can escape the $web by quoting it in single quotes ' - or use azure cli for that:
az storage blob sync -c $web –account-name melcherit -s “C:\agent_work1\s\public”

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Rodrigo Werlang [e-mail]
Sent: Wednesday, July 17, 2019 9:00 PM
To: microsoft/azure-pipelines-tasks
Cc: Max Melcher; Mention
Subject: Re: [microsoft/azure-pipelines-tasks] AzureFileCopy support AzCopy (v10) sync command (#9157)

I'm having the same issue publishing a reactjs to a storage account.
The problem to workaround is that PowerShell does not support using a container named as $web, because $ is treated as invalid.

Any other workarounds to delete the blob before using the AzureFileCopy?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/9157?email_source=notifications&email_token=AAH3LNYXIXOKYA4ZIXWC6ODP75T4ZA5CNFSM4GL6IEH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2GIKSY#issuecomment-512525643, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAH3LN272GZZTPFTUSET433P75T4ZANCNFSM4GL6IEHQ.

[dpolivy]

@rajatagrawal-dev Are there plans for a v4 for this? Another feature, which is very important for me, is the ability to set the Cache-Control header on the files being uploaded to blob storage. It seems that is not possible with AzCopy v8, but it is available in AzCopy v10.

Is there any other way to set the Cache-Control header as part of the current Azure File Copy Task?

[mikegoatly]

This issue seems to have become a general request for v10 AzCopy support. Related to this I've found that v8 doesn't seem to publish the same ContentTypes as v10 - specifically .wasm files should have the 'application/wasm' content type. Publishing using AzCopy with Storage Explorer the content type is set correctly, but using AzureFileCopyV3 it is set to application/octet-stream. (This is even with using the /SetContentType flag - other files have their content types set correctly, e.g. HTML/JS files)

[jbuffin]

Adding my support for v10 AzCopy. I need to set cache-control so that I can use the cache feature of Front Door for my static website hosted in blob.

[VitalyDukhota]

Adding my support for v10 AzCopy. I need to set cache-control so that I can use the cache feature of Front Door for my static website hosted in blob.

Same please - I need cache control from v10.

[RoopeshNair]

We are planning to pick this up in next couple of sprints. Please stay tuned, will update once we have more details

[akunaatrium]

How's it going?

[20shivangi]

@eelkram We have already added support for AzCopy V10 in new version of AzureFileCopy task. Please check the task AzureFileCopy with major version V4.

[albernhagen]

@20shivangi By default, it's forcing usage of copy with this task. How can I change it to sync? Here's what's running for me (I tried passing sync to the additional arguments and it didn't work): "AzCopy\AzCopy.exe" copy "D:\a\1\dist" "https://myblobstorageurl" sync

[ninallam]

@albernhagen As of now we have added support only for the copy command to the latest version of the Azure File Copy task. Sync mode is not supported in the task. In AzCopyV10 the sync has a different syntax and cannot be sent as an additional argument to the copy command.
Refer to the below documentation for AzCopy sync syntax
https://docs.microsoft.com/azure/storage/common/storage-ref-azcopy-sync?toc=%2fazure%2fstorage%2fblobs%2ftoc.json

[albernhagen]

@ninallam thank you, that makes sense. Is there any plan to create a AzFileSync task in the near future that utilizes the sync command instead?

[ninallam]

@albernhagen We have the AzCopy sync mode in our backlogs but haven’t been able to reach to it yet. I would recommend posting this feature ask in our developer community feedback so that others also can vote for it and we can prioritize it better.