Velero: Support backing up and restoring Helm workloads

[thdrnsdk]

Hello!

I am one of the users who is using ark now.
The ability to back up objects in kubernetes is great.
By the way, our system distributes the application using helm and uses helm history as the lifecycle of the application.

Do you plan to do ark backup and restore in helm's deploy unit instead of kubernetes object?

[nrb]

Hi @thdrnsdk. There is an Ark Helm chart available at https://github.com/helm/charts/tree/master/stable/ark, but it is not currently maintained by the Ark team.

Ark also doesn't have any logic to handle Helm resources differently than other Kubernetes objects at the moment.

Can you provide a bit more detail on what behavior you're looking for?

[thdrnsdk]

oh it seems to be misunderstood. I'm sorry my poor English.

i know ark is deployed by helm chart.

my question is

Is there any way to backup and restore k8s objects deployed using helm?

for example i'm deployed "nginx" use for helm chart.

$ helm install --name=nginx nginx

and

$ helm ls

show the list of helm deploy. next update object use for helm cli

$ helm update nginx XXXXX

in this state when i backup the objects, and restore the other cluster it can not tell how the nginx that I restored was deployed and updated with helm.

Well I do not know if my question is understandable. If you do not know, I'll show you a detailed example later.

thank you

[ncdc]

My understanding is there is a ConfigMap that Helm uses to keep track of the things it's deployed. For the time being, you would need to make sure that your backup includes the relevant Helm ConfigMaps, and that they're included when restoring. I don't think we've tried this, so we'd be interested to hear how it goes if you attempt it.

[thdrnsdk]

hello @ncdc
i'm already test it.

The namespace with tiller and the namespace to backup are different, so we could not create one "backups.ark.heptio.com"CRD object.

If I have same namespace of the tiller that I want to backup, it is not a problem, but as you know, deploying a tiller in every namespace seems a bit wasted.

So I asked if there was any other way.

[ncdc]

The namespace with tiller and the namespace to backup are different, so we could not create one "backups.ark.heptio.com"CRD object.

A single Ark Backup can back up multiple namespaces. If you're using the cli: ark backup create my-backup --include-namespaces myapp,tiller.

However, maybe we can try to be more efficient:

1. Have the Ark server know about a default namespace for tiller (ark server --tiller-namespace foo)
2. When backing up, allow the user to override the default tiller namespace (ark backup create my-backup --tiller-namespace bar)
3. When backing up, if Ark can identify an item as belonging to a Helm chart deployment (is there a label for this?), also back up the appropriate ConfigMap from the tiller namespace.
4. When restoring, restore the ConfigMap along with all the other items (pods, etc).

Do you think something like this could work?

[Evesy]

Hi, This is something we've been thinking about too. We (generally) namespace our applications and in the event we need to restore one, we need to restore the relevant ConfigMap's for the release too.

Helm does label the ConfigMaps as per the below:

    MODIFIED_AT: "1540987831"
    NAME: platform-unsee
    OWNER: TILLER
    STATUS: SUPERSEDED
    VERSION: "3"

When we've had to do this previously for an individual application we've restored in two phases:
ark restore create --from-backup=<> --include-namespaces <APP_NAMESPACE>
ark restore create --from-backup=<> --include-resources configmaps -l NAME=<APP_NAME>

One option is to ensure the resources created via your helm release all have the matching NAME label applied, but a multi-selector restore would also be useful (i.e. This namespace && these labels in a different namespace) -- I think that extends what label selectors are capable of though.

We haven't come across this issue on the Backup side of things since we do full cluster backups, but I can see it definitely being sought after in the future

[ncdc]

@Evesy does Helm automatically label all the items it creates (pods, services, etc) with a common label, or is that entirely up to the chart itself?

[Evesy]

@ncdc Unfortunately not. There's a bit of a convention of having the below labels, but even those don't align with Helm ConfigMap's:

chart:
release:
heritage:

[ncdc]

Not aligning with the ConfigMap is fine, assuming the ConfigMap has a naming convention we can discern... But it sounds like there's no way to 100% guarantee that we could automate collecting all resources created by a chart + its associated ConfigMap. Is that accurate?

[Evesy]

Not that I can see, either way you'd be relying on naming conventions to match resources with a helm release.
The only way to discern exactly what was created by a helm release would be to inspect the currently deployed release versions ConfigMap to identify which resources are managed by that Helm release.

[ncdc]

Does the ConfigMap list everything that Helm created for the chart?

[Evesy]

It'll include all 'standard' resources that Helm has created, however CRD's created by crd-install hooks, as well as any actions taken with other pre/post install hooks won't be in there, so even that isn't reliable :(

[Dennor]

I actually wrote a plugin which does some basic helm release backup including resources created by hooks. The only thing I'm unsure is how to tell velero to that ResourceIdentifier is cluster scoped (for example CRD).

[skriss]

@Dennor I believe you just need to leave Namespace blank - does that not work?

[skriss]

btw, awesome on the plugin! I'll check it out :)

[Dennor]

Yeah, scratch my question. The first thing I tried was blank Namespace but apparently I messed something up somewhere else. Now it works.

Thanks.

[brysonshepherd]

@skriss @ncdc have you had a chance to look at @Dennor 's Plugin? Because Helm is so widely used, it would be nice to have the native ability to backup the helm state/config as well.

[oskapt]

I actually wrote a plugin which does some basic helm release backup including resources created by hooks. The only thing I'm unsure is how to tell velero to that ResourceIdentifier is cluster scoped (for example CRD).

As of 2019-09-22 that plugin is not compatible with Velero 1.x. There is an open issue addressing this, and the plugin maintainer said that he'll eventually get around to updating it.

An error occurred: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

EDIT 2019-09-23: added link to issue

[mjschmidt]

@oskapt can you link the issue please!?!

[oskapt]

@mjschmidt edited and added link

[mjschmidt]

Thank you!

[empinator]

did anybody come across a solution for backing up/restoring helm based installations with velero?
I'd love to see my restored application showing up with 'helm ls' and therefore being able to do updates the helm-way.

I did see one method where one would only restore the PVC and then install the app via helm, causing the new installation to use the existing volumes. This eventually works, but only if you don't have init-containers involved.

[runzexia]

https://github.com/runzexia/velero-plugin-helm
i make @Dennor 's plugin could work with Velero 1.x, you may want to try this