Rear: BTRFS and LXD restore problems

[stephen-d-hill]

• ReaR version ("/usr/sbin/rear -V"):
  Relax-and-Recover 2.3 / Git

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):

LSB Version:    core-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
OUTPUT_URL=sftp://pi:raspberry@omv/export/backup/
BACKUP=NETFS
#BACKUP_OPTIONS="soft"
BACKUP_URL=sshfs://pi@omv/export/backup/
BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/var/lib/lxd/*' '/home/*' '/shared_data/*' '/shared_data2/*' '/var/w$
EXCLUDE_MOUNTPOINTS=( '/var/lib/lxd/shmounts' '/var/lib/lxd/devlxd' '/shared_data' '/shared_data2' )
EXCLUDE_DEVS
# Workarounds needed for NFS...
PROGS=( "${PROGS[@]}" showmount mount.nfs umount.nfs )
MODULES=( "${MODULES[@]}" nfs )
PRE_RECOVERY_SCRIPT="systemctl start rpcbind.target || rpcbind &"
CLONE_ALL_USERS_GROUPS=yes

• Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
  mkrescue - Thinkserver TS140 Intel Xeon E3-1226 v3 (4) @ 3.700GHz, baremetal
  recover - VirtualBoxVM

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
  x86_64

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  BIOS, GRUB

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  local disk

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 931.5G  0 disk
├─sda1   8:1    0     1M  0 part
├─sda2   8:2    0   256M  0 part
├─sda3   8:3    0     1G  0 part /boot
├─sda4   8:4    0   899G  0 part /
└─sda5   8:5    0  31.3G  0 part [SWAP]
sdb      8:16   0 931.5G  0 disk
sdc      8:32   0   3.7T  0 disk
sdd      8:48   0   3.7T  0 disk /shared_data
sde      8:64   0   3.7T  0 disk /shared_data2
sdf      8:80   0   1.8T  0 disk
sr0     11:0    1  1024M  0 rom  ](url)

• Description of the issue (ideally so that others can reproduce it):
  Restore fails when trying to create LXD subvolumes.

ERROR: target path already exists: /mnt/local//var/lib/lxd/storage-pools/default

~$ sudo btrfs subvolume show /
/
        Name:                   <FS_TREE>
        UUID:                   -
        Parent UUID:            -
        Received UUID:          -
        Creation time:          -
        Subvolume ID:           5
        Generation:             4432883
        Gen at creation:        0
        Parent ID:              0
        Top level ID:           0
        Flags:                  -
        Snapshot(s):
                                var/lib/lxd/storage-pools/default/containers/lms
                                var/lib/lxd/storage-pools/default/containers/serviio
                                var/lib/lxd/storage-pools/default/containers/plexserver
                                var/lib/lxd/storage-pools/default
                                var/lib/lxd/storage-pools/default/containers
                                var/lib/lxd/storage-pools/default/snapshots
                                var/lib/lxd/storage-pools/default/images
                                var/lib/lxd/storage-pools/default/custom
                                var/lib/lxd/storage-pools/default/containers/drawio/rootfs/var/lib/docker/btrfs/subvolumes/c9ef627fa63935a459a81012c277edd81238477ceb2c1655f8c118db1aa1fcbc
                                var/lib/lxd/storage-pools/default/containers/drawio/rootfs/var/lib/docker/btrfs/subvolumes/6d5b919fe59170e48fa31053fc654037bc8db18c8411f05998bee60f45641eac
                                var/lib/lxd/storage-pools/default/containers/pigallery/rootfs/var/lib/docker/btrfs/subvolumes/aa2390d77147e501b7a6eff5329ac9c2bebd439fb7ebe1eb9826b68203b893bc

• Workaround, if any:
  Not sure what to try

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  rear-hill.log
  diskrestore.sh.txt

[stephen-d-hill]

Note that I have also tried v2.5:

$ rear -V
Relax-and-Recover 2.5 / 2019-05-10

but has same problem.

[stephen-d-hill]

Note that I don't actually want to backup the LXD containers ;-)

[jsmeix]

@stephen-d-hill
could you also attach your var/lib/rear/layout/disklayout.conf file
that you got after "rear mkbackup/mkrescue"?

[jsmeix]

@stephen-d-hill
when you have LXD related entries in your disklayout.conf file
that you don't need to be recreated you could manually comment out
those entries before you run "rear recover".

[stephen-d-hill]

@stephen-d-hill
when you have LXD related entries in your disklayout.conf file
that you don't need to be recreated you could manually comment out
those entries before you run "rear recover".

Yes, I'll try that.
Is there some way I can get it to exclude these subvolumes, via local.conf?

[stephen-d-hill]

@stephen-d-hill
could you also attach your var/lib/rear/layout/disklayout.conf file
that you got after "rear mkbackup/mkrescue"?

disklayout.conf.txt

[jsmeix]

You may have a look at the section "Excluding components" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc

But I did not test if or how that works to exclude things like

btrfsnormalsubvol /dev/sda4 / 715 var/lib/lxd/storage-pools/default
btrfsnormalsubvol /dev/sda4 / 716 var/lib/lxd/storage-pools/default/containers
...

[stephen-d-hill]

You may have a look at the section "Excluding components" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc

But I did not test if or how that works to exclude things like

btrfsnormalsubvol /dev/sda4 / 715 var/lib/lxd/storage-pools/default
btrfsnormalsubvol /dev/sda4 / 716 var/lib/lxd/storage-pools/default/containers
...

I tried adding the following to /etc/rear/local.conf:
EXCLUDE_RECREATE+=( '/mnt/local//var/lib/lxd/storage-pools/default' '/mnt/local//var/lib/lxd/storage-pools/default/containers' )
But got same error.

[jsmeix]

According to the section "Excluding components" in
https://github.com/rear/rear/blob/master/doc/user-guide/06-layout-configuration.adoc
the syntax should be more like

EXCLUDE_RECREATE+=( "btrfsnormalsubvol:<something>" )

where <something> matches an entry in the disklayout.conf file.
But again: I never tested if excluding btrfs subvolumes is supported.

[stephen-d-hill]

So, I tried to get clever, as I was getting bored of waiting for mkbackup to complete (1.5h); as all I wanted was to see the disklayout rear was creating.
So I ran rear -v help, and saw the layoutonly option :thinking:. So I ran that on the server I wanted to backup, thinking it would just create the disk layout file. Big mistake...it tried to _create_ the partitions. It fails at some point, but not before wiping out my partition table :grimacing:. Shame it didn't spot that I had partitions mounted on that device before doing that :frowning_face:.
To attempt to recover the situation, I've run testdisk analysis which found my partitions again and rewrote the partition table. I daren't reboot :worried:, although looking at /sys/block/sda and sfdisk -l /dev/sda output they match up, so in theory the partition table is back they way it was, apart from the partition numbers are different (my /etc/fstab table uses UUIDs, and the kernel shows that sda1/sda2 didn't exist before). rear though complains about non-contiguous partitions, so I can't now perform a backup (given the current situation, probably not a bad thing) without a reboot.

[stephen-d-hill]

Attempted a restore of the backup with the LXD problem...
If I comment out the lxd directories during a restore, then the restore works fine into a virtualbox VM :grinning:. Using the power of virtual machine snapshots, I broke this VM the same way I broke my server, using the layoutonly option. I then proceeded to use testdisk to fix the partition table and then rebooted; was still broken as grub had also been lost. I retried using a snapshot, but this time also performed grub-install /dev/sda; this time the VM booted up fine, so have now done same on my bare-metal server :grinning:.

[jsmeix]

@stephen-d-hill
regarding your
https://github.com/rear/rear/issues/2387#issuecomment-626303944

Ouch!!
It must not happen that possibly destructive workflows
that are intended to be run only inside the ReaR recovery system
could be also run by accident from inside the normal/original system.

I did
https://github.com/rear/rear/pull/2395
to fix that.

[jsmeix]

Regarding your
https://github.com/rear/rear/issues/2387#issuecomment-626319777
therein in particular "comment out the lxd directories" [in disklayout.conf]:

I think you could automate such things by using PRE_RECOVERY_SCRIPT
https://github.com/rear/rear/blob/master/usr/share/rear/conf/default.conf#L3033

[jsmeix]

@stephen-d-hill
regarding your
https://github.com/rear/rear/issues/2387#issuecomment-626303944
therein "rear though complains about non-contiguous partitions"

According to your
https://github.com/rear/rear/issues/2387#issue-612884083
you use ReaR version 2.3.

In our recent ReaR GitHub master code we have
support for creation of non consecutive partitions, see
https://github.com/rear/rear/pull/2081

See the section
"Testing current ReaR upstream GitHub master code" in
https://en.opensuse.org/SDB:Disaster_Recovery

[jsmeix]

@stephen-d-hill
regarding your
https://github.com/rear/rear/issues/2387#issuecomment-626303944
therein "bored of waiting for mkbackup to complete"

Use rear mkrescue to only create the ReaR recovery system
without making a backup of the files of your system.

Use rear savelayout to only create the disklayout.conf file anew.

[github-actions[bot]]

Stale issue message