Velero: Unable to delete backup if cloud resources have already been deleted (velero v1.5.1, similar to #308)

[titou10titou10]

As issue #308 is closed, I opened this new one as I'm facing the same issue but with v1.5.1

The blobs have already been deleted in the cloud and velero is unable to delete the backup

velero backup get
velero-s-24h-20200831015039  Completed   0   12   2020-08-31 01:32:51 -0400 EDT   17h       default            <none>
velero-s-24h-20200826022813  Deleting    0   0    2020-08-26 01:47:20 -0400 EDT   4d ago    default            <none>

Velero logs (Split for readability):

level=error msg="Error in syncHandler, re-adding item to queue" controller=backup-deletion error="error downloading backup: error copying Backup to temp file: 
rpc error: code = Unknown desc = storage: service returned error: StatusCode=404, ErrorCode=BlobNotFound, ErrorMessage=The specified blob does not exist.\nRequestId:75a99b8d-f01e-004e-175c-963ab1000000\nTime:2020-09-29T12:30:26.7728994Z, 
RequestInitiated=Tue, 29 Sep 2020 12:30:26 GMT, RequestId=75a99b8d-f01e-004e-175c-963ab1000000, API Version=2018-03-28, QueryParameterName=, QueryParameterValue=" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_controller.go:558" 
error.function=github.com/vmware-tanzu/velero/pkg/controller.downloadToTempFile key=velero/velero-s-24h-20200826022813-766pb logSource="pkg/controller/generic_controller.go:140"

[zubron]

Thanks for reporting this, @titou10titou10!

Just to confirm, I can reproduce this locally with a minio setup by deleting the directory for a backup in the configured bucket.

[zubron]

It looks like this behaviour was broken in #2815. As part of performing DeleteItemActions, the tarball for the backup is downloaded which will fail if the resources have been deleted from the cloud storage.

[zubron]

I'm going to start on this and add it to the 1.5.2 milestone given that it's been reported a couple of times now, and is a regression from the 1.4.2 behaviour.

[zubron]

I've taken a look at this. It's trivial to skip the downloading of the tarball if there are no DeleteItemActions to perform but it raises a further question of what the behaviour should be in the failure case (where there _are_ actions to perform but the associated backup resources have been deleted and the actions can't be performed because the tarball can't be downloaded).

Should the deletion of the backup continue and simply log that the DeleteItemActions couldn't be performed? Should the deletion (and its associated request) "fail"?

@nrb As the author for the design, I'd really appreciate your input here :)

[dsu-igeek]

Since we don't know what items would need to get deleted, I think it's fine at this point to not call the DeleteItemActions. This may result in orphaned objects, but we don't have a general cleanup for orphaned objects at the moment.

[nrb]

Agreeing with @dsu-igeek here - since we have no idea what we'd match the DeleteItemActions against, I think we should log that the tarball was missing, then skip any DeleteItemAction plugins that are present.

[survivant]

in my case the backup were created with a invalid storage location
````
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
cm-vault-backup FailedValidation 0 0 29d default app=test
cm-vault-backup2 FailedValidation 0 0 29d default app=test

time="2020-11-12T15:08:40Z" level=warning msg="The specified default backup location named \"default\" was not found; for convenience, be sure to create one or make another backup location that is available the default" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:161"
time="2020-11-12T15:09:40Z" level=info msg="Checking for existing backup locations ready to be verified; there needs to be at least 1 backup location available" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:58"
time="2020-11-12T15:09:40Z" level=warning msg="The specified default backup location named \"default\" was not found; for convenience, be sure to create one or make another backup location that is available the default" controller=backupstoragelocation logSource="pkg/controller/backupstoragelocation_controller.go:161"
````

[survivant]

root@test-pcl114:~/openebs-test# velero version
Client:
Version: v1.5.2
Git commit: e115e5a191b1fdb5d379b62a35916115e77124a4
Server:
Version: v1.5.2

[Zeka13]

at least what is workaround ??

[zubron]

@Zeka13 As a workaround, you can delete the Backup item directly using kubectl delete backups.velero.io -n velero <backupname>. However, deleting the item directly will mean that it skips any processing that would normally be performed by Velero (such as deleting other objects in storage associated with the backup). This may not be an issue for you, depending on the state of the backup you wish to delete, but it's something to keep in mind. Where possible, we strongly recommend you use the standard velero delete command for deleting backups.

[charlspjohn]

@Zeka13 As a workaround, you can delete the Backup item directly using kubectl delete backups.velero.io -n velero <backupname>. However, deleting the item directly will mean that it skips any processing that would normally be performed by Velero (such as deleting other objects in storage associated with the backup). This may not be an issue for you, depending on the state of the backup you wish to delete, but it's something to keep in mind. Where possible, we strongly recommend you use the standard velero delete command for deleting backups.

Will this method delete all the pvc snapshots associated with the backup which is getting deleted ?