Velero: can pods created by a DaemonSet be excluded from backups?

[mattnworb]

We recently had a need to restore a cluster from a backup, which worked pretty great overall (thank you for Ark!).

We noticed after the restore that, for a DaemonSet we run for fluentd, twice as many pods existed than we expected - one set of pods that were created by the DaemonSet, and another set of fluentd pods that were not attached to the DaemonSet but did have the label ark-restore: <name of backup> indicating that those Pods were restored from the backup.

Is it possible to configure Ark to exclude Pods created by DaemonSets from being backed up, since the post-restore DaemonSet will manage to create one pod-per-node on its own?

[ncdc]

@mattnworb sorry for the delay in responding! We'll take a look and get back to you as soon as we can. Thanks for your patience!

[ncdc]

@mattnworb

I looked at the DaemonSet controller code in Kubernetes. If I'm reading it correctly, it's doing this:

If the pod has a controllerRef where the owner's kind is DaemonSet, and the owner's UID doesn't match the UID of the DaemonSet in question, it ignores that pod. I'm guessing the assumption is that "some other" DaemonSet is in charge.

If you're doing a true recovery, where Ark is restoring both the DaemonSet and its pod(s), we'll end up in the situation you reported. Ark isn't allowed to restore the original UID when we create the DaemonSet (it's assigned by Kubernetes), so there's no way the pod's controllerRef can point at the UID of the newly-restored DaemonSet, unless we restore the pod after we restore the DaemonSet. However, if we take that approach, it's possible/likely that the DaemonSet will create its own pod(s) around the same time Ark is creating them, ending up with extra pods that ultimately will get deleted by the DaemonSet controller.

It might make more sense to recognize there are some things, like in this situation, where Ark can't really "win" when recreating resources. We could maybe take a look at letting the DaemonSet create its pod(s), then find some way for Ark to update them and carry over any labels & annotations that are present in the backup. I'm not entirely sure how this would work, though - what if the backup has 5 pods for a DaemonSet - how do we map a pod in the backup to a pod in the new cluster?

This definitely needs some more thinking...

[mattnworb]

Hi @ncdc thanks for the reply and looking into this!

The operation in question that led to me creating this issue was in fact a true recovery - Ark restored both the DaemonSet and all of the Pods in the (original) cluster that were created by the DaemonSet.

What we noticed after the Ark restore was complete was that we ended up with 2x as many pods for this DaemonSet as we expected - the N pods that the restored DaemonSet created, one per node, and the N pods that had been in the backup. The latter half of these pods were not being managed by the (restored) DaemonSet as you pointed out.

In the case I am thinking of, I would be happy if Ark could be configured to not bother to backup pods that were created by DaemonSets - I'd be confident that a restored DaemonSet would create the necessary pods after being restored.

[ncdc]

@mattnworb what I'm struggling with the most is figuring out the best way to configure this along with all the other things we'll discover that also need tweaking. A simple fix for this could be to add a new --include-daemonset-pods flag (default to false) and have that map to a field on the Restore spec with the same name (we'd add it to the Restore instead of the Backup because our preference is to back up everything, but restore only what's necessary). I'm a bit worried that we'll find other similar needs, and I'm not sure if adding a new flag/field for each situation is appropriate, or if there's some better, more holistic approach. We're working on finalizing a stable API for 1.0, so we need to think through these issues carefully.

Once the rest of the team is back from the Thanksgiving holiday next week, we can have a more thorough discussion. Thanks again for your patience!

[eleanor-millman]

Gnarly and hard and we will wait until we hear more users feeling this pain before we re-open.

[sseago]

FYI, as a workaround you could create a pod RestoreItemAction that excludes pods with DaemonSet OwnerReferences, or depending on your use case, perhaps exclude all pods with owner references, since the owning DaemonSet, Deployment, etc. will recreate the pod on restore.

[sseago]

I've observed the same problem with DaemonSet pods if the pod isn't excluded from restore.