Velero: Request for new plugin kind to run custom delete logic while deletion of backup.

[ashish-amarnath]

Describe the problem/challenge you have
Resources created during backup and restore of items by their corresponding ItemAction plugins will not get cleaned up unless the backup deletion controller is updated to perform the necessary cleanup.

Describe the solution you'd like
There should be a new plugin kind that runs on backup deletion. Implementation of this plugin can be used to perform cleanup/house keeping for resources created during backup and restore of resources by their corresponding ItemAction plugins.

cc @nrb @carlisia @skriss

[nrb]

Resources created during backup and restore of items by their corresponding ItemAction plugins will not get cleaned up unless the backup deletion controller is updated to perform the necessary cleanup.

To be very clear here - these are resources explicitly tied to and created by the Velero backups themselves, such as CSI VolumeSnapshots. Resources that are simply discovered via the API server, such as the pod -> pvc -> pv relationship are not covered by this.

At the moment, I'm not aware of any plugins that do this besides the Velero CSI plugins, so we're building it into core. That being said, I can see this being useful in the long term, especially as upstream data protection efforts grow. So I think we should track this but rank its priority low for the moment.

[carlisia]

unless the backup deletion controller is updated to perform the necessary cleanup

To clarify: we have the option of updating the deletion controller to handle this clean up, or to create a new plugin kind? If so, I would prefer to update the deletion controller.

[agill17]

I have a use case where I have developed a custom backup plugin that runs some arbitrary logic and creates a snapshot of an external resource. Unfortunately when velero backups expire or is deleted by hand, the external snapshot does not get deleted. I would like to have an extension point that runs some cleanup logic when backup CR is cleaned up.

[nrb]

@carlisia Yeah, the option was to add a new plugin type.

[nrb]

I'm starting to reverse my thinking here, because of the following:

• Other folks want to implement something like our approach to the CSI plugins, but they don't necessarily have the option to alter the core Velero code base.
• Other segments of the Velero core logic are implemented within plugins - see the pod backup item action, CRD version remapping, service account secret adjusting, storage class remapping, and a host of others.
• Moving to deletion plugins that are invoked at Backup deletion time would allow us to move the in-tree deletion logic that removes Restore, VolumeSnapshot, and VolumeSnapshotContent CRs into their own, isolated segments such that the backup deletion controller can get smaller (and maybe make kubebuilder/controller-runtime migration easier?)

@vmware-tanzu/velero-maintainers What do you think?

[nrb]

@agill17 Are the things you want to delete Kubernetes Custom Resources, or something else?

I ask because another idea I had was to perhaps register a set of CRDs to clean up at deletion time, looking for CRs labeled with the Velero Backup's name. This would cover deleting Custom Resources, but not things like data stored in non-k8s systems.

This behavior could also perhaps be implemented as one of the deletion plugins that takes a ConfigMap with the CRDs in it.

[ashish-amarnath]

+1 _upvoting my own issue_

[vishnuitta]

In case of CSI model of plugins i.e., BackupItemAction, it makes sense to have extension to perform cleanup during deletion.
OpenEBS being volume snapshotter, DeleteSnapshot API is good enough to perform cleanup activities during Backup deletion.

This behavior could also perhaps be implemented as one of the deletion plugins that takes a ConfigMap with the CRDs in it.

This as well sounds like a generic solution.

[nrb]

Yeah, for VolumeSnapshotter implementations, I think it makes sense.

In situations where a VolumeSnapshotter doesn't fit, though, I think we need to provide something a bit more robust.

[skriss]

Adding deletion plugins seems like a reasonable thing to me.

If it's just CRs that someone is interested in having deleted, they may be able to get away with setting owner refs to the Velero backup and taking advantage of Kubernetes cascading deletion.

But, there are definitely scenarios where that won't be sufficient.

[nrb]

Yeah, setting the owner refs isn't a bad idea in any case, though existing API objects wouldn't work that way.

In some ways this is a little cart-before-the-horse, as the use cases are a little slim, but allowing for invocation of custom code at deletion or (or maybe even prior to?) Velero Backups makes the system more flexible overall.

[nrb]

If we use a configmap, it will have to be synced between clusters.