Azure-cli: Add option to fully synchronise blobs on azure with local folder content

[SteveALee]

Id like a command to syncronise a container etc to a local directory tree while setting the contentType for all files. Options exist to upload files to plobs in the CLI. This could be an option to the azure blob upload-batch

That means it should remove blobs that no longer appear as files in the local directory tree. I have found no such option or other portable CLI tool to do this (not even the old AzCopy). this is critical in applications including a static web site.

The main use case is using blob storage for a static web site and updateing automatically from script for continuous deployment. During rapid iterations the structure can vary and cruft will get left around if blobs are only added. That has a cost impact. This use case also require #1436

With out this option the CD will have to delete all the blobs before uploading. That will require getting a list of blobs, or simply recreating the container, which will lose meta data.

There is also minor security issue with the blobs served over http. Unwanted resources may be addressable when they should not be there at all.

[troydai]

In brief what you're looking for is a mirror option, which purges the none existing files while updates the newly updated file. Am I right?

[Tragetaschen]

At least for me, these two options would be awesome. I'm currently storing about 4.5 GiB of blobs and when I need to update, only a small fraction of these actually have changed. Currently, I'm just uploading them all again and don't purge the ones that don't exist anymore.

[DouglasHeriot]

With the new Azure static website hosting, this would be even more helpful. I'd like to script my deployment process, and have it only upload files that have changed, and delete files that no longer need to be there. https://azure.microsoft.com/en-us/blog/azure-storage-static-web-hosting-public-preview/

@troydai I see this was added to a previous sprint but never completed. Should it be put back on the backlog, or even better, scheduled to a future sprint?

[atrauzzi]

Yeah, this is super important! Would love to see this made available as soon as can be done.

See the AWS sync command, specifically with the --delete option for inspiration.

@troydai

[troydai]

@atrauzzi thank you for 'at' me, however, I'm not working at Microsoft now. @tjprescott and @yugangw-msft will take over this. You're in good hands.

[williexu]

@atrauzzi for now, please try out azcopy v10's azcopy sync command.

Unfortunately, the CLI has not been optimized for large batch uploads/downloads and any implementation we make using the current python storage sdk will likely suffer from performance hits.
We are, however, looking into adding this functionality and I thank you for the feedback.

[atrauzzi]

@williexu - I don't have azcopy in my toolchain at this stage as I'm not on Windows and don't want to have .net core as a dependency of my non-core builds. This is a situation where az needs to be a one-stop-shop.

If it isn't optimized for such an activity, that strikes me as a deficiency in its design. AWS CLI has no issues offering this, I'd hate to be forced into admitting their design is _better_! 😉

Very much hope you guys can get this working!

[williexu]

@atrauzzi the new version of azcopy is written in Golang and there are executables for non-windows machines: https://github.com/Azure/azure-storage-azcopy#installation

[atrauzzi]

Nice, although I still think az should incorporate all functionality. It's very confusing to have multiple CLI tools when one is promising to be the place for everything.

[williexu]

We will be taking a dependency on azcopy for this feature through our storage-preview extension soon. :)

[limingu]

The sync command is in our storage-preview extension now.

Try that like:
az storage azcopy blob sync
-c containerName
--account-name accountName
-s directory/path
-d sync/path

[yugangw-msft]

I believe with today's release, the command should be az storage blob sync and it should be very fast since we are integrating azcopy

[limingu]

The sync command is in CLI.

Try that like:
az storage blob sync
-c containerName
--account-name accountName
-s directory/path
-d sync/path