Velero: Support restoring a PV into the same cluster

[ncdc]

Forked from #189

Scenario:

• PVC in namespace _N1_, bound to PV _P1_
• Create backup (snapshotting the PV)
• Restore backup, remapping namespace _N1_ to _N2_
• Have the PVC in _N2_ reference a new PV _P2_, created from _P1_'s snapshot

Currently, this is not possible, because Ark does not change the name of the PV when restoring it. The result is 2 PVCs both trying to claim the same (original) PV _P1_.

Questions:

• Should this be done automatically?
• Should this be optional?
• What sort of controls/flags should we expose to the user?
• How do we cover the 80-90% use case?

cc @evaldasou

[ncdc]

@rosskukulinski would you find this useful?

[dgoodwin]

Our usage is planned to always delete a namespace after backup. (as we're using it to archive it) As such I don't think we'd hit a scenario where we backed up, but still had a live namespace using the PV.

However restoring with a remapped namespace and preserving the PV would definitely be useful, we don't plan to do this for a bit but it is on the radar. (I forget if Ark can handle remapped namespace with PV snapshots if nothing is still using the PV)

[rosskukulinski]

@ncdc reading through #189, yes I think this would be a useful (and expected IMO) feature for CI/CD and testing use-cases.

As a more specific example, this feature request enables what you describe in the ephemeral ci testing env w/ ark blog post but supports this behavior within a single cluster.

[evaldasou]

wow, awesome! So as @rosskukulinski mentioned , it looks like in Your blog post @ncdc, ephemeral ci testing env w/ ark - it is supported to do this type of restore from one cluster to another? (without deleting original PV/PVC?). I am going to test this!

Regarding new feature, I think there should be optional flag , so we can select to re-create PVC/PV or not. Something like _--restore-volumes-recreate_ (to recreate PVC/PV) flag.

[ncdc]

Current thinking is ark restore create NAME --clone-volumes would take each PV p in the backup, restore as PV p', and ensure any PVC being restored that references p is updated to reference p'.

[ncdc]

cc @jbeda on UX

[jbeda]

It might be worthwhile to think about this in terms of scenarios and the flags that will usually exist together. 90% of the time that users remap namespaces they'll want to rename/clone PVs too. We don't want to have people flip a zillion flags to do the common case.

I'm reminded of the flags for rsync:

-a, --archive
    This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and
    want to preserve almost everything.

Another analogy/pattern: git. While git isn't the best, most usable thing but there is the idea of plumbing vs. porcelain. https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain. Think about what is the underlying granular toolkit (plumbing) and what is the usable outer shell (porcelain).

[temujin9]

Is this on a near-term roadmap, or should I be finding a workaround?

[temujin9]

(Also, are you interested in outside contribution, or are you tackling this internally?)

[ncdc]

We’d love to get to this soon if possible. We definitely could use some
help with UX ideas. And we always welcome contributions!

On Fri, Apr 20, 2018 at 3:40 PM Nathaniel Eliot [e-mail]
wrote:

(Also, are you interested in outside contribution, or are you tackling
this internally?)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/heptio/ark/issues/192#issuecomment-383200977, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAABYhS5ZV7SO1T3TCdUQx908gy7Fxuaks5tqjmugaJpZM4QW2E5
.

[rosskukulinski]

Following up on this: I think this is a critical feature. Key use-case is enabling duplication of namespaces within the same cluster.

This should not be done automatically [for cross-cluster restores or for DR, should keep the PV names the same].

Suggestion: --unique-pv. If there are other features/flags required to enable expected behavior for namespace duplication, we can create a short-code that enables multiple flags (like how rsync does).

[agolomoodysaada]

Are there any plans on implementing this? I'm happy to submit a PR with a few guidelines from authors.
Thanks for the great work you guys are doing ❤️

[ncdc]

@agolomoodysaada we do hope to be able to include this feature. We'll be doing some more project planning in the next week or two, and hopefully we'll have time to discuss this one. We'd also be thrilled if you wanted to contribute a PR for this. But before starting, I do think this is a feature in particular that needs a solid design for UI/UX. Please read the comments in this issue, and let us know if you have any ideas. Thanks!

[jengo]

Any updates on this ?

[skriss]

@jengo not much change to report. It would be awesome if someone wanted to work on this - happy to collaborate!

[arianitu]

We want to take a shot at implementing this. @skriss @ncdc any tips on where we should start in the codebase?

We want to be able to restore N1 to N2 for creating a production environment from a staging environment, etc.

We would probably only focus on GKE first since that's what we use. I am unsure if the cloud environment matters for the implementation.

[skriss]

@arianitu I'll look at this some more today and see if we can come up with a reasonable approach here.

[arianitu]

@skriss any update on this, have you found an approach, is it a good idea for us to start looking at implementation for this?

[skriss]

@arianitu I've been looking into it. Definitely interested in your view on what the right UX is. Should this be automatic behavior when using the --namespace-mappings flag? Should it be a separate flag on velero restore create, e.g. --rename-pvs? Something else?

[robertgates55]

The 3 use cases where you’d use the —namespace-mapping that I see:

• Cross-cluster copy (with namespace name change)
• in-cluster fork (new namespace that looks the same as another)
• in-cluster move (rename namespace)

Only the fork scenario requires new PV names I think - so an additional —rename-pvs flag would work nicely for that case?

[agolomoodysaada]

I like your first suggestion @skriss .

velero backup create --include-namespaces ns1 ns1-backup
# in the same cluster
velero restore create --namespace-mappings ns1:ns2 --from-backup ns1-backup

The problem with same-namespace restores and the --rename-pvs flag is that you would need to rename more than just PVs; deployment names, Statefulset names, etc... I think it's okay to release an early implementation of this feature that requires the --namespace-mappings to be implemented for all PVs that are to be restored. It could fail if any PV restore does not have a namespace mapping.

Unless... unless we can do something like --prefix-names restored- which would rename all resources.

velero restore create --prefix-names "restored-" --from-backup ns1-backup
# mydeployment -> restored-mydeployment
# mypvc -> restored-mypvc
# mystatefulset -> restored-mystatefulset

[arianitu]

@skriss I think the behaviour should be automatic if you're going from one namespace to another. In most cases, if you are using a PVC in one namespace, the next namespace is not going to work correctly since almost always you're going to run into an issue where containers will complain about the PVC already being mounted.

Would there be a specific reason to having a —rename-pvs flag? Also, —rename-pvs makes it sound like the old PVs are no longer going to be used when I think we want new PVs from a snapshot to be used and we would keep the old PVs around. (is this possible with Velero?)

[skriss]

@robertgates55 re: https://github.com/heptio/velero/issues/192#issuecomment-514771036 -- you make a good point that it's only necessary to rename the PV in one of those scenarios; however, do you see a problem if we also rename the PV for the other scenarios? Given that PV names are usually system-generated random IDs, I wonder how much it matters if the name gets changed on restore (as long as its related PVC gets updated as well).

One option is to have this be automatic behavior when using the --namespace-mappings flag, but to only rename the PV if there's already one in the target cluster with the current name.

[agolomoodysaada]

I'm not sure if it's applicable here. But cloning a volume could be done using the new PVC DataSource (aka Volume Cloning) introduced in Kubernetes 1.15 as an alpha feature.
My understanding is that you'll be able to clone PVCs with PVs that have the same disk contents. Perhaps a tight integration with this feature might resolve this issue?

More info: https://github.com/kubernetes/enhancements/pull/1147/files

[robertgates55]

@skriss I think that renaming the pv only if the existing ones are already in use in the cluster would be ideal

[arianitu]

Thank you for your work @skriss, looking forward to testing this, is there a version I can test on our systems?

[skriss]

Yes! you could try using the gcr.io/heptio-images/velero:master image in your deployment. Note that this is a dev build, so I wouldn't recommend using it production.

You shouldn't need an updated client, as the changes are all server-side.

[arianitu]

@skriss Sorry swinging back on this after a long time, but I am on:

Client:
    Version: v1.3.1
    Git commit: -
Server:
    Version: v1.2.0

But I still get

  Type     Reason         Age    From                         Message
  ----     ------         ----   ----                         -------
  Warning  ClaimMisbound  5m26s  persistentvolume-controller  Two claims are bound to the same volume, this one is bound incorrectly

Do you know which Velero version supports this?

[arianitu]

Also to describe what we are doing:

1. We have a namespace A that we create a backup with.
2. We restore backup to namespace B.
3. Namespace A still has original PV and PVC

Is this use case supported? We are basically looking for a namespace clone for testing purposes.