Azure-cli: Cannot delete full directory structure from Directory in File Share

[dsparkplug]

Is your feature request related to a problem? Please describe.

The az storage file upload-batch command can be used to upload a full directory structure to an Azure File Share or Directory.

The az storage file delete-batch command can be used to recursively delete all the files in an Azure File Share or Directory. It does however leave all the empty subfolders.

There does not seem to be a way of deleting all the empty subfolders - unless each subfolder is deleted individually. This requires prior knowledge of the folder names.

Describe the solution you'd like

The az storage file delete-batch command should have an option to delete empty subfolders.
Alternatively, or additionally, the az storage directory delete should have an option to delete all subfolders.

Describe alternatives you've considered

Note that is is possible to delete the whole file share using az storage share delete --name sparkysfileshare just not a specific directory within a file share.

Additional context

Here's an example workflow:

> az storage directory create --name sparkysfolder --share-name sparkysfileshare 
{
  "created": true
}

> az storage file upload-batch --destination sparkysfileshare --source I:\temp\test --destination-path sparkysfolder --pattern * 

uploading I:\temp\test\test1.txt
Finished[#############################################################]  100.0000%
uploading I:\temp\test\subfolder\test2.txt
Finished[#############################################################]  100.0000%
[
  "https://vsoftfilestorageac.file.core.windows.net/sparkysfileshare/sparkysfolder/test1.txt",
  "https://vsoftfilestorageac.file.core.windows.net/sparkysfileshare/sparkysfolder/subfolder/test2.txt"
]

> az storage file list --share-name sparkysfileshare --path sparkysfolder

[
  {
    "metadata": null,
    "name": "test1.txt",
    "properties": {
      "contentLength": 4,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": null,
        "contentType": null
      },
      "copy": {
        "completionTime": null,
        "id": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "file"
  },
  {
    "metadata": null,
    "name": "subfolder",
    "properties": {
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "dir"
  }
]

> az storage file delete-batch --source sparkysfileshare --pattern sparkysfolder\* --account-name vsoftfilestorageac 

> az storage file list --share-name sparkysfileshare --path sparkysfolder

[
  {
    "metadata": null,
    "name": "subfolder",
    "properties": {
      "etag": null,
      "lastModified": null,
      "serverEncrypted": null
    },
    "type": "dir"
  }
]

> az storage directory delete --name sparkysfolder --share-name sparkysfileshare 

The command failed with an unexpected error. Here is the traceback:

The specified directory is not empty. ErrorCode: DirectoryNotEmpty
<?xml version="1.0" encoding="utf-8"?><Error><Code>DirectoryNotEmpty</Code><Message>The specified directory is not empty.
RequestId:17d1cf8b-001a-000b-7fab-f584e4000000
Time:2019-04-18T05:59:17.7350008Z</Message></Error>
Traceback (most recent call last):
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\knack\knack\cli.py", line 206, in invoke
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 326, in execute
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 384, in _run_jobs_serially
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 377, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\six\six.py", line 693, in reraise
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 354, in _run_job
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\commands\__init__.py", line 145, in __call__
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-cli-core\azure\cli\core\__init__.py", line 451, in default_command_handler
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\file\fileservice.py", line 1035, in delete_directory
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\_error.py", line 97, in _dont_fail_not_exist
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\file\fileservice.py", line 1032, in delete_directory
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 381, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 306, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\storageclient.py", line 292, in _perform_request
  File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-z6u9vbmn\azure-multiapi-storage\azure\multiapi\storage\v2018_03_28\common\_error.py", line 115, in _http_error_handler
azure.common.AzureConflictHttpError: The specified directory is not empty. ErrorCode: DirectoryNotEmpty
<?xml version="1.0" encoding="utf-8"?><Error><Code>DirectoryNotEmpty</Code><Message>The specified directory is not empty.
RequestId:17d1cf8b-001a-000b-7fab-f584e4000000
Time:2019-04-18T05:59:17.7350008Z</Message></Error>

[limingu]

We will add the option to 'az storage directory delete' to delete all empty subfolders.

[ryansantaana]

Is this feature going to be added soon? I have a CI/CD workflow within Azure Pipelines where I need to delete a directory and upload a new one with each release, and this feature would be nice to have.

[Juliehzl]

@dsparkplug You need to use az storage directory delete --name sparkysfolder/subfolder --share-name sparkysfileshare to delete empty folder. But we will improve it in S162.

[dsparkplug]

@Juliehzl

@dsparkplug You need to use az storage directory delete --name sparkysfolder/subfolder --share-name sparkysfileshare to delete empty folder. But we will improve it in S162.

If you look at my example workflow above, you will see that is one of the commands that I did use. The issue was not with deleting a single empty folder - the issue was with deleting a full directory structure.

I look forward to seeing the improvements in S162

[ekelmans]

you gotta be kidding me....

So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

[ChandraSM]

Modified version of the above script to delete a sub-directory under main directory inside a file share:

function RemoveFileDir ([Microsoft.Azure.Storage.File.CloudFileDirectory] $dir, [Microsoft.Azure.Commands.Common.Authentication.Abstractions.IStorageContext] $ctx)
{
$filelist = Get-AzStorageFile -Directory $dir
foreach ($f in $filelist)
{
if ($f.GetType().Name -eq "CloudFileDirectory")
{
RemoveFileDir $f $ctx #Calling the same unction again. This is recursion.
}
else
{
Remove-AzStorageFile -File $f
}
}
Remove-AzStorageDirectory -Directory $dir
}

define varibales

$StorageAccountName = "storageaccountname"
$StorageAccountKey = "storageaccountkey"
$AzShare = "filesharename"
$AzDirectory = "rootdirectoryunderfileshare"

create primary region storage context

$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
$ctx.ToString()

Check for Share Existence

$S = Get-AzStorageShare -Context $ctx -ErrorAction SilentlyContinue|Where-Object {$_.Name -eq $AzShare}

Check for directory and delete the sub directory which is not needed-

$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name

if ($d.Name -notcontains $AzDirectory)
{
Write-Host "directory is not present; no action to be performed"
}
else
{
#Write-Host "Entered"
$dir1 = Get-AzStorageFile -Share $S -Path $AzDirectory
$dir2 = Get-AzStorageFile -Directory $dir1 | Where-Object Name -EQ "subdirectoryunderroot"
$dir3 = Get-AzStorageFile -Directory $dir2 | Where-Object Name -EQ "foldertodeleteundersubdirectory"
RemoveFileDir $dir3 $ctx
}

[zhoxing-ms]

I think it is necessary for the Service to implement the function of deleting the sub empty folder, the main reasons are as follows:

1. If the CLI implements the deletion of all subfolders by itself, it will recursively query and delete all subdirectories. If the number of folders is large, the performance will be slow due to too many requests, and if the folder is deep, the stack will overflow
2. Customers who invoke the rest API may also have a need to delete all empty subfolders under a folder.

[msftbot[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

[CondormanFr]

The command :
$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7
It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

[BryanSoltis]

The command :
$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7
It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

I'm getting this same error when developing a PowerShell Azure Function. Any update on this issue?

[ekelmans]

Nope,

No reply from MS on this issue yet ☹

Met vriendelijke groet,

Theo Ekelmans
Sr. MS-SQL DBA

Ringwade 1
3439 LM Nieuwegein
T [phone]
M [phone]
http://www.ordina.nl/ www.ordina.nl

Eerstelijn MS SQL Team
Werkdagen tussen 08:00 – 18:00
T [phone]
E [e-mail] [email protected]

7x24 klanten: 0800-0231841

From: Bryan Soltis [e-mail]
Sent: Monday, 22 June 2020 17:03
To: Azure/azure-cli [e-mail]
Cc: Theo Ekelmans [e-mail]; Comment [e-mail]
Subject: Re: [Azure/azure-cli] Cannot delete full directory structure from Directory in File Share (#9141)

The command :
$d = Get-AzStorageFile -Share $S -ErrorAction SilentlyContinue|select Name does not work with powershell 7
It answers :

Get-AzStorageFile: Cannot bind parameter 'Share'. Cannot convert the "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" value of type "Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageFileShare" to type "Microsoft.Azure.Storage.File.CloudFileShare"

.

I'm getting this same error when developing a PowerShell Azure Function. Any update on this issue?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cli/issues/9141#issuecomment-647575028 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AIBPH4VJBDQQ5TWUZL5MMJTRX5XC5ANCNFSM4HG2AFYQ . https://github.com/notifications/beacon/AIBPH4VHV3G3QXTRWYAMX5LRX5XC5A5CNFSM4HG2AFY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOE2MTL5A.gif

[edwardwu-servian]

you gotta be kidding me....

So i have a tree with 40.000+ folders and over 3.000.000 files, and those need to be deleted on a per item basis ?

@ekelmans try az storage file delete-batch -s=$SHARE_NAME --account-name=$ACCOUNT_NAME. i've recently bump into issue with az storage directory delete (I am using terraform - under the hood it's using Azure REST API )

[janegilring]

@ekelmans I currently experience the same when developing an Azure Function running PowerShell 7 for exporting some files to an Azure File Share. Did you have any progress on your case?

Update: The following alternate technique seems to work

$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $BackupStorageAccountResourceGroupName -Name $BackupStorageAccountName)[0].Value
$Context = New-AzStorageContext -StorageAccountName $BackupStorageAccountName -StorageAccountKey $StorageAccountKey


Get-ChildItem -Recurse -File -Path Temp:/backup/ | ForEach-Object {
    $path = '/Backups/KeyVault/' + (Split-Path $_.FullName -Leaf)
    Set-AzStorageFileContent -ShareName $BackupStorageAccountFileShareName -Source $_.FullName -Path $path -Force -Context $Context
    $path
}

[ekelmans]

@ekelmans I currently experience the same when developing an Azure Function running PowerShell 7 for exporting some files to an Azure File Share. Did you have any progress on your case?

Update: The following alternate technique seems to work

$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $BackupStorageAccountResourceGroupName -Name $BackupStorageAccountName)[0].Value
$Context = New-AzStorageContext -StorageAccountName $BackupStorageAccountName -StorageAccountKey $StorageAccountKey


Get-ChildItem -Recurse -File -Path Temp:/backup/ | ForEach-Object {
    $path = '/Backups/KeyVault/' + (Split-Path $_.FullName -Leaf)
    Set-AzStorageFileContent -ShareName $BackupStorageAccountFileShareName -Source $_.FullName -Path $path -Force -Context $Context
    $path
}

Hi J,

No answer just yet, though the dev team said they have added it to the todo list, but gave no timeframe for a new release.

The problem with the foreach-object loop is that is is Very slow if you have millions of files in thousands of folders. BUT... for now it helps to get rid of the smallest folders

Thanks fo sharing :)

[janegilring]

I see. Did you try the new -Parallel switch on Foreach-Object in PowerShell 7?

[ekelmans]

WOW.... No,

I never knew the existence of such a switch, I will give it a try, thanks again :)

[bpossolo]

+1 please add a way to delete empty folder tree using the azure-cli
or better yet, update az storage file delete-batch to delete the folders as well.... that's what i expected it would do