Rear: Failed to create recovery system (too small ESP on USB device - no space for .../EFI/BOOT/initrd.cgz)

[gaia]

Relax-and-Recover (ReaR) Issue Template

• ReaR version:
  Relax-and-Recover 2.4

• OS version:
  Debian 10 (buster)

• ReaR configuration:

OUTPUT=USB
BACKUP=BORG
USB_DEVICE=/dev/disk/by-label/REAR-000
###
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER="/boot/efi/EFI/grubx64.efi"
###
SSH_UNPROTECTED_PRIVATE_KEYS="no"
SSH_FILES="avoid_sensitive_files"
###
EXCLUDE_MOUNTPOINTS=( '/zfs_containers2' '/mnt/storage' '/mnt/containers' '/mnt/usb' '/mnt/bindmounts' '/mnt/storage/swapfile' )
EXCLUDE_MD=( '/dev/md1' )
EXCLUDE_VG=( '/dev/mapper/lvmthin0' '/dev/mapper/lvmthin1' )
###
BORGBACKUP_REPO="/my_borg_backup"
BORGBACKUP_UMASK="0002"
BORGBACKUP_PRUNE_DAILY=7
BORGBACKUP_ENC_TYPE="keyfile"
BORGBACKUP_COMPRESSION="lz4"
BORGBACKUP_SHOW_PROGRESS="yes"
BORGBACKUP_SHOW_STATS="yes"
export BORG_PASSPHRASE='__________'
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
###
PROGS_BORG=( )
BACKUP_PROG_INCLUDE=('/etc/pve/' )
COPY_AS_IS_BORG=( '/root/.config/borg/keys/' )
COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" 'home/*/.cache/*' 'var/cache/*' 'var/tmp/*' 'dev/*' 'media/*' 'proc/*' 'sys/*' 'tmp/*' 'var/run/*' 'var/lock/*' 'var/lib/apt/lists/*' '/var/cache/apt/*' )
COPY_AS_IS=( $SHARE_DIR $VAR_DIR )

• Hardware:
  Bare Metal Xeon 9th gen

• System architecture:
  x64

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  UEFI + Grub2

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  Spinning Rust + NVME disks

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

• Description of the issue:

2020-11-24 04:30:34.623833405 ======================
2020-11-24 04:30:34.624841108 Running 'backup' stage
2020-11-24 04:30:34.626411395 ======================
2020-11-24 04:30:35.002765598 Including backup/default/005_valid_backup_methods.sh
2020-11-24 04:30:35.210903671 Including backup/default/010_pre_backup_script.sh
2020-11-24 04:30:35.261015771 Including backup/BORG/default/100_get_suffix.sh
2020-11-24 04:30:35.321998787 Including backup/BORG/default/250_mount_usb.sh
2020-11-24 04:30:35.335243934 Mounting with 'mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.60xzwC41QJxGTjm/borg_backup'
mount: /dev/sde2 mounted on /tmp/rear.60xzwC41QJxGTjm/borg_backup.
2020-11-24 04:30:35.654866407 Including backup/BORG/default/400_create_include_exclude_files.sh
2020-11-24 04:30:35.839167065 Including backup/BORG/default/500_make_backup.sh
2020-11-24 04:30:35.887096791 Creating archive rear_270 in repository /my_borg_backup
2020-11-24 04:44:10.968795129 ERROR: Failed to create backup
==== Stack trace ====
Trace 0: /usr/sbin/rear:543 main
Trace 1: /usr/share/rear/lib/mkbackuponly-workflow.sh:16 WORKFLOW_mkbackuponly
Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source
Trace 4: /usr/share/rear/backup/BORG/default/500_make_backup.sh:34 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:371 StopIfError
Message: Failed to create backup

• Workaround, if any:
  N/A, or at least IDH. It worked under this config for a while, then it stopped.

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  N/A

[gozora]

From the logs you've provided I can see that your USB Borg repository already contains some previous backups (archives), What I mean is that you've already run some successful backups prior this error. Is that correct?
If so, can it be that your USB disk is experiencing some IO errors/HW problems?
If not, can you please provide full log from rear -d -D mkbackup session ?

Thanks

V.

[gaia]

Relax-and-Recover 2.4 / Git Using log file: /var/log/rear/rear.log Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1) Creating disk layout Excluding RAID /dev/md1. Excluding Volume Group /dev/mapper/____. Excluding Volume Group /dev/mapper/____ Using guessed bootloader 'EFI' (found in first bytes on /dev/sda) Creating root filesystem layout Skipping 'vmbr1': not bound to any physical interface. Cannot include keyboard mappings (no keymaps default directory '') Using '/boot/efi/EFI/proxmox/grubx64.efi' as UEFI bootloader file Copying logfile /var/log/rear/rear.log into initramfs as '/tmp/rear-partial-2020-11-25T13:39:29-08:00.log' Copying files and directories Copying binaries and libraries Copying kernel modules Copying all files in /lib*/firmware/ Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression Created initrd.cgz with gzip default compression (205115772 bytes) in 24 seconds ERROR: Could not copy /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz to /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz Aborting due to an error, check /var/log/rear/rear.log for details Exiting rear mkbackup (PID 12639) and its descendant processes Running exit tasks You should also rm -Rf /tmp/rear.JsEmyhSq6taUJk5 Terminated

Note I altered the actual block device names in EXCLUDE_VG, that's why it does not match the other logs.

[gozora]

Can you attach /var/log/rear/rear-node2.log ?

[gaia]

the pword to the zip file is your username

thanks

[gozora]

You need bigger ESP on your USB device:

Excerpt from the log:
... ++ cp -p -v /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz '/tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz' -> '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz' cp: error writing '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz': No space left on device ...

ReaR controls size of ESP partition by USB_UEFI_PART_SIZE variable. Be aware that after updating of USB_UEFI_PART_SIZE, you need to reformat your USB storage (rear format ...), which will obviously cause DATA LOSS.

V.

[gdha]

@gozora @jsmeix perhaps it would be a nice idea to have something like https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh to scan for errors?

[gozora]

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from _96_call_for_action_after_preview.sh_ on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

[gdha]

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from _96_call_for_action_after_preview.sh_ on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

That is indeed true, but out of space can be search out of array of the most common errors for example.

[jsmeix]

ReaR 2.4 is rather old and the terminal output in
https://github.com/rear/rear/issues/2526#issuecomment-733962038
shows that ReaR 2.4. does not have our current ReaR
enhanced Error function behaviour that always shows
Some latest log messages since the last called script ..., cf.
https://github.com/rear/rear/blob/master/usr/share/rear/lib/_input-output-functions.sh#L557

I implemented this enhanced Error function behaviour via
https://github.com/rear/rear/commit/2420688530e84cf5fbee37b715d9e22bc8ba7f2e
and
https://github.com/rear/rear/pull/1877
in ReaR 2.5, see doc/rear-release-notes.txt

Now the Error function shows some last messages of the last sourced
script to the user (issues #1877 #1875)

https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L1419

For an example how current ReaR Error function behaviour
looks like on the user's terminal see the rear -v mkrescue output at
https://github.com/rear/rear/issues/2524#issue-749678000

I assume if @gaia would have used current ReaR
there would have been some more useful output about the actual error
on the terminal where "rear mkrescue/mkbackup" was run
and then this issue report would perhaps have not been needed
because the root cause would have been directly visible to the user.

I think this issue can be closed and no further attention is needed.

[jsmeix]

@gdha
only for the fun of it:
Your script
https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh
is a "no-go" because it contains the forbidden word WARNING, cf.
https://schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
;-))

[jsmeix]

@gozora
thank you for your time requesting ReaR debug log files
and inspecting them to show the user the root cause!