Azure-cli: `Failed precondition` when trying to batch delete blobs

[ElvenSpellmaker]

Describe the bug
When running:

    az storage blob delete-batch \
        --account-name "$AZURE_ACCOUNT_NAME" \
        --account-key "$AZURE_ACCOUNT_KEY" \
        --source "$CONTAINER" \
        --if-unmodified-since "$DATE"

I get:

Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
Failed precondition
[]

To Reproduce
Run

    az storage blob delete-batch \
        --account-name "$AZURE_ACCOUNT_NAME" \
        --account-key "$AZURE_ACCOUNT_KEY" \
        --source "$CONTAINER" \
        --if-unmodified-since "$DATE"

Expected behavior
The blobs to be deleted.

Environment summary
Installed in a virtualenv through pip.

Happens on both a CentOS TeamCity agent and locally on Cygwin.

[williexu]

@ElvenSpellmaker it looks like all your blobs failed the --if-unmodified-since "$DATE" precondition

[ElvenSpellmaker]

@williexu Hey, this works if I delete each blob individually by working out the dates manually, see: https://github.com/ElvenSpellmaker/AzureOldBlobDelete/pull/2

[ElvenSpellmaker]

Basically if I work out the date manually then blobs clearly need deleting, so as far as I can tell it's a server-side bug.

[williexu]

@ElvenSpellmaker seems to work fine for me:

(env36) Willies-MacBook-Pro:azure-cli williexu$ az storage blob delete-batch -s wilxcontainer --account-name wilxstorage --if-unmodified-since 2018-06-12T23:34Z
Failed precondition
Failed precondition
[]
(env36) Willies-MacBook-Pro:azure-cli williexu$ az storage blob delete-batch -s wilxcontainer --account-name wilxstorage --if-unmodified-since 2018-06-12T23:36Z
[
  null,
  null
]

Can you try once more?
Keep in mind that the dates you provide must be in UTC.
Also make sure that the date you provide in the parameter is later than the 'lastModified' property on your blobs if you expect them to be deleted.

[ElvenSpellmaker]

@williexu I'll see if I can find an account to try it on, but is that seriously the expected response format, an array of nulls if it's deleted something, plus a load of Failed precondition to the terminal with no blob names or anything if it fails to meet the date?

That output is so unusable for anything. 😂

[williexu]

@ElvenSpellmaker can you raise a separate issue for the messages displayed?
On that note, Failed Precondition messages do seem like unnecessary bloat, I may change that to 'debug' output or remove them entirely.

[tjprescott]

@williexu you can just repurpose this issue. We should probably do some smart collecting of errors instead for these batch cases. There are other issues around this theme (confirm on a delete with --ids, for example). The error should look something like:

X of X files not deleted because of 'Failed Precondition'

Or something similar. I don't think it would be useful for anyone's debug stream to get that output either.

[ElvenSpellmaker]

Hey @tjprescott, most commands from az may be used in scripts and parsed appropriately, however this endpoint doesn't even return (just) valid JSON (or whatever format is chosen, like tsv), and the JSON it does return is just nulls in an array. (And the only usefulness of that is being able to count the number of deleted entities.)

Showing the failed precondition messages at all in the normal (non-debug) stream will just cause parsing problems.

In my case it'd be nice to know the names of the blobs which were deleted.

[tjprescott]

Such messages are printed to stderr so they will not interfere with any parsing. This command in particular is a CLI convenience command, not the result of a single API call, so we have full control over the output format (which is sounds like needs some work).

[williexu]

I think @tjprescott's suggestion to print out the number of failures to stderr is the right move.

In terms of output, we have to consider breaking users scripts if we do make a change; currently, most delete commands that utilize the --ids parameter will output the same result.
As most delete operations in the CLI are supposed to return no output, whatever change we might make will likely not output the names of the deleted blobs.

@ElvenSpellmaker, for now, you can use the storage blob list command to find out the effects from delete-batch. In addition, did your original problem of using the --if-unmodified-since parameter get resolved? If so, I can edit this issue to better reflect the message output.

[highbass]

this is happening with me currently and i am using azure-cli version 2.0.33 ... blobs dont actually get deleted ...

my command is

az storage blob delete-batch --source inbox --verbose --if-unmodified-since `gdate -u -d "14 days ago" '+%Y-%m-%dT%H:%MZ'`

when i do something like

az storage blob delete -c inbox -n blob-name --if-unmodified-since `gdate -u -d "14 days ago" '+%Y-%m-%dT%H:%MZ'`

than it works perfectly fine ...

[williexu]

@highbass can you update to the latest version of the CLI and try again?

In addition, even with the version you have, delete-batch should call the same method as delete with the same parameters.

When you retry, please make sure you do a blob list to check what blobs were deleted; it may be that some blobs failed the precondition, but the batch-command still deleted the individual blob you tried to delete the second time around. delete will succeed if the blob had already been deleted.

[williexu]

Raised a separate issue for the logging issue: https://github.com/Azure/azure-cli/issues/6960

[highbass]

@williexu I upgraded to the latest version and still the same thing... now i wrote a script to iterate over blob list and delete if it meets the date condition and its working properly via that method. I have like millions of blob files so its been running for 24 hours and still not close to the end.

[ElvenSpellmaker]

@highbass I also gave up and wrote a script (scripts) to loop through them and delete individually: https://github.com/ElvenSpellmaker/AzureOldBlobDelete

[williexu]

@highbass I just successfully ran the following command:

az storage blob delete-batch --account-name wilxstoragebatch -s wilxcontainer --if-unmodified-since `date -u -d "10 minutes ago" '+%Y-%m-%dT%H:%MZ'` --debug

Have you tried doing a blob list after to see if any of your blobs were deleted (it's possible that some failed while others succeeded)?.
In addition, if you can, try this on a smaller container, which you can populate with a smaller number of blobs and run the command with --debug so we can see what is happening.

I can't reproduce a problem with the delete-batch command.

[williexu]

Adding an example for the scenario with: https://github.com/Azure/azure-cli/pull/7153/files#diff-32882f7652ab67bdbdbf7815216079faR350

[williexu]

Closing as the output issue with failed precondition has been resolved in the current behavior for delete-batch