Lxd: Allow for snapshot/restore of custom storage volumes

[nakatadim]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04.3
• The output of "lxc info" or if that fails:
  
  
  
  • Kernel version: 4.10.0-38-generic
  
  
  • LXC version: 2.1.1
  
  
  • LXD version:2.20
  
  
  • Storage backend in use: thin lvm
  
  

Issue description

I have a container with mounted additional volume from another storage pool. I've noticed that "lxc snapshot mycontainer" command makes snapshot only of the root volume but not on the additionally mounted one.

Maybe that's by design because i found the following description in func containerSnapshotsPost:

/*
* snapshot is a three step operation:
* 1. choose a new name
* 2. copy the database info over
* 3. copy over the rootfs
*/
Do you think it's a good idea to have an option on the attached volume or on the snapshot command itself to make snapshots on all volumes used, not only on rootfs?

Steps to reproduce

1. "lxc snapshot mycontainer" where mycontainer has attached additional volumes

[stgraber]

So I don't think we want/can snapshot all storage volumes automatically. That's mostly because storage volumes can come from storage pools that are using a different storage driver and not all storage drivers allow for the same snapshot/restore operations.

The main issue here is with ZFS as ZFS doesn't allow rolling back to anything but the latest snapshot.
If you want to rollback to an earlier snapshot you must delete any snapshot that happened after it.

That's annoying but not a huge problem when you only have one container using a particular storage volume, but if you had more than one container attached to the storage volume, rollingback any of the containers would now break rollingback of any of the others.

[stgraber]

I think our best bet here is to add the ability to snapshot custom storage volumes separately from the containers themselves. So effectively getting new "lxc storage volume snapshot", "lxc storage volume restore" commands added to drive this.

Having a "lxc snapshot -R" flag to do recursive snapshotting may then be reasonable so long as we make sure our documentation makes the impact of this clear to users.

[rldleblanc]

I'm trying to migrate from LXC to LXD and I have a number of containers that have SSD and HDD storage. I'm trying to figure out the best way to convert that and still work with my btrfs send/receive backup scripts, but use the LXD snapshot function as much as possible. None of these containers share the data with each other and having the ability to go back to a point in time with all volumes would be ideal.

Is it possible to add a disk device from a storage pool or does it have to be an absolute path? It seems that storage pool would allow better container migration.

Like VMs, I would suggest that volumes from storage pools be considered local to only the container unless special configuration is done to specify that it is shared, then prevent snapshotting/migration of the shared volumes.

I have TBs of data to migrate to LXD and using cp reflink=always, I can prevent re-backing up all the data, I just need to understand the best way forward. Maybe for now, I have LXD snapshot the base volume, then manually btrfs snap the data volume, then btrfs send/receive both volumes off-site using my backup scripts. Sound right?

[rldleblanc]

Okay, I found lxc storage volume create and lxc storage volume attach which answers one of my questions. This method still doesn't snapshot the added volume when the container is snapshotted.

[stgraber]

Custom volumes will never get snapshoted when the containers they're attached to are. That's because they may be attached to multiple containers and some of the storage backends don't make it possible to delete intermediary snapshots.

What we'll do is allow you to snapshots custom volumes separately from the containers they're attached to.

[brauner]

I think we'll start with:

diff --git a/lxd/storage_volumes_snapshot.go b/lxd/storage_volumes_snapshot.go
new file mode 100644
index 000000000..d3793b8e5
--- /dev/null
+++ b/lxd/storage_volumes_snapshot.go
@@ -0,0 +1,20 @@
+package main
+
+import (
+       "fmt"
+       "net/http"
+)
+
+func storagePoolVolumeSnapshotsTypeGet(d *Daemon, r *http.Request) Response {
+       return NotImplemented(fmt.Errorf("Retrieving storage pool volume snapshots is not implemented"))
+}
+
+func storagePoolVolumeSnapshotsTypePost(d *Daemon, r *http.Request) Response {
+       return NotImplemented(fmt.Errorf("Creating storage pool volume snapshots is not implemented"))
+}
+
+func storagePoolVolumeSnapshotsTypeDelete(d *Daemon, r *http.Request) Response {
+       return NotImplemented(fmt.Errorf("Deleting storage pool volume snapshots is not implemented"))
+}
+
+var storagePoolVolumeSnapshotsTypeCmd = Command{name: "storage-pools/{pool}/volumes/{type}/snapshots", post: storagePoolVolumeSnapshotsTypePost, get: storagePoolVolumeSnapshotsTypeGet, delete: storagePoolVolumeSnapshotsTypeDelete}

[stgraber]

Well, that's actually wrong :)

var storagePoolVolumeSnapshotsTypeCmd = Command{name: "storage-pools/{pool}/volumes/{type}/snapshots", post: storagePoolVolumeSnapshotsTypePost, get: storagePoolVolumeSnapshotsTypeGet}
var storagePoolVolumeSnapshotTypeCmd = Command{name: "storage-pools/{pool}/volumes/{type}/snapshots/{name}", post: storagePoolVolumeSnapshotTypePost, get: storagePoolVolumeSnapshotTypeGet, delete: storagePoolVolumeSnapshotTypeDelete}

Should be more like it.

[brauner]

The one remaining thing to settle is how we want the on-disk layout to look like without restructuring everything. Right now we have:

root@conventiont|/var/lib/lxd/storage-pools/default/custom
> ls -al ../
total 20
drwxr-xr-x 1 root root   62 May 30 13:00 .
drwx--x--x 6 root root 4096 May 31 12:50 ..
drwxr-xr-x 1 root root   46 Jul  9 12:58 containers
drwxr-xr-x 1 root root    8 Jul 12 11:44 custom
drwxr-xr-x 1 root root  256 Jul 11 21:54 images
drwxr-xr-x 1 root root    4 Jul 12 11:35 snapshots

I suggest to not add an additional folder or restructure everything on-disk but rather just prefix all storage volume snapshots. Such that you end up with <parent-storage-volume>-snapshot-<snapshot-name> but I'm open to other suggestions.

[stgraber]

That's an option though I'm not sure what would be so hard with doing something like:

• Move snapshots to be containers-snapshots
• Create custom-snapshots
• Put custom snapshots under custom-snapshots/<custom volume name>/<snapshot name>

[brauner]

Not the difficulty that bothered me but rather the ugliness. But both solutions are so whatever. :)

[ThomasLeister]

I have a similar use case like @rldleblanc : My containers should use SSD storage (ZFS) for root and HDD storage (also ZFS) for bigger files. HDD storage is attached as a separate volume. I'd like to have an easy way to snapshot both storage volumes :-)

A dedicated command for additional / custom volumes would be sufficient. (as proposed by @stgraber )

[stgraber]

@ThomasLeister it's coming, we have a pull request up for that which still has a bit of work left on it, but should be there in 3.6.