Rear: mkinitrd on Centos fails due missing /var/tmp

[gozora]

• rear version (/usr/sbin/rear -V): 2.2
• OS version (cat /etc/rear/os.conf or lsb_release -a): CentOS Linux release 7.3.1611 (Core)
• rear configuration files (cat /etc/rear/site.conf or cat /etc/rear/local.conf):

BACKUP=NETFS
OUTPUT=ISO

BACKUP_URL=nfs://node1/mnt/backup
OUTPUT_URL=nfs://node1/mnt/backup/iso

GRUB_RESCUE=n

COPY_AS_IS=( ${COPY_AS_IS[@]} /sbin/sysctl /etc/sysctl.conf /sbin/vconfig /sbin/if* /etc/sysconfig/network /sbin/shutdown.wrap /usr/bin/strace /usr/sbin/dmsetup /usr/bin/ipcrm /usr/bin/ipcs)

BACKUP_PROG_EXCLUDE=( ${BACKUP_PROG_EXCLUDE[@]} '/mnt/*' '/media/*' '/var/tmp' )

• Are you using legacy BIOS or UEFI boot: Legacy BIOS
• Brief description of the issue: mkinitrd fails when whole _/var/tmp_ directory is excluded from backup
• Work-around, if any: Keep directory by using exclude pattern _/var/tmp/*_ instead of _/var/tmp_

This problem originates from mkinitrd (dracut) behavior which uses _/var/tmp_ as default value for --tmpdir. If _tmpdir_ is missing during finalize phase, initrd recreation fails with following message:

2017-08-25 07:22:48.921465572 Running mkinitrd...
mktemp: failed to create directory via template '/var/tmp/dracut.XXXXXX': No such file or directory
dracut: mktemp -p '/var/tmp/' -d -t dracut.XXXXXX failed.
2017-08-25 07:22:49.072362263 WARNING:
Failed to create initrd for kernel version '3.10.0-514.el7.x86_64'.

To fix this, I'd go for following simple code in _finalize/Fedora/i386/550_rebuild_initramfs.sh_ (maybe _SUSE_LINUX/i386/550_rebuild_initramfs.sh_ suffer by similar behavior ...)

if [ ! -d /var/tmp ]; then
  mkdir /var/tmp
fi

V.

[schlomo]

@gozora good catch. We have scripts for that sort of thing:

$ git ls-files "*create*missing*"
usr/share/rear/restore/SESAM/default/900_create_missing_directories.sh
usr/share/rear/restore/SUSE_LINUX/910_create_missing_directories.sh
usr/share/rear/restore/default/900_create_missing_directories.sh

Maybe you can add /var/tmp to one of them?

[gozora]

Thanks @schlomo, so something like this would be suitable?

+++ /usr/share/rear/restore/default/900_create_missing_directories.sh   2017-08-25 10:37:22.111606674 +0000
@@ -26,4 +26,9 @@ else
     done
 fi
 chmod 1777 tmp
+
+if [ ! -d var/tmp ]; then
+    mkdir -p var/tmp
+    chmod 1777 var/tmp
+fi
 popd >/dev/null

V.

[schlomo]

Yes, as simple as that is probably enough.

[gozora]

With #1456 merged, this issue can be closed.

[jsmeix]

@gozora
many thanks for finding and fixing all those smaller
but annoying issues.

FYI:
I think a system without a /var/tmp/ directory
can be considered to be broken nowadays
so that strictly speaking it is the admin's task
to make a backup that contains all required stuff.
I am not a FHS expert but I think the directories listed in
https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard
could be expected to "just exist".

[schlomo]

@jsmeix I think that this is a corner case or grey area: Typical backup software assumes that the restore always happens on top of a system that was installed the traditional way, so that for the backup software it is safe to assume that those directories are present. From the perspective of the backup software there is therefore no need to include such directories in a full backup.

Bottom line is, I think it is the duty of ReaR to make sure that these special directories really do exist.

[jsmeix]

@schlomo
many thanks for your explanation!
With that I changed my mind and state now that
"rear recover" shoud create a system with the
"usual directory structure" (whatever that actually means).

I think it would be not correct if "rear recover" just creates
the usual (or some minimal) FHS directory structure from scratch
because that may not match what there was on the original system.

Therefore I think during "rear mkrescue" the "usual directory structure"
of the original system should be saved into a ReaR config file,
e.g. var/lib/rear/layout/directorylayout.conf
from which "rear recover" could recreate it.

I think the problem would be special permission settings
like ACLs and such stuff.

Can perhaps 'tar' be used to backup only a directory structure
without the files? Then we might use 'tar' for that?

Or something like 'cp -a' only for directories without files?

[schlomo]

@jsmeix I don't think that we need to create a generic solution. We already store and recreate the mount points as lots of backup tools omit them. I would keep things simple and add common directories that users report as missing to MOUNTPOINTS_TO_RESTORE in default.conf.

@gozora after looking at the code again I realized that there is an even simpler solution:

1. Add var/tmp to MOUNTPOINTS_TO_RESTORE in default.conf
2. Add var/tmp to the chmod 1777 call already present here

That way we will treat /tmp and /var/tmp the same.

[jsmeix]

Could we then - by the way - also
rename MOUNTPOINTS_TO_RESTORE
into DIRECTORIES_TO_CREATE
so that its name tells what it actually is
and then it matches better to the script name
that does it 900_create_missing_directories.sh

Currently MOUNTPOINTS_TO_RESTORE is used only in
conf/default.conf
restore/default/900_create_missing_directories.sh
verify/TSM/default/400_verify_tsm.sh

[jsmeix]

I could do that next week it you agree.

[jsmeix]

@gozora
just wait - I will prepare a pull request right now...
(that can be rejected if not wanted)...

[schlomo]

Thanks guys, I also think that we should rename the variable. And we should probably update the TSM script to add its list of TSM excluded mountpoints to that list instead of replacing it.

[gozora]

@jsmeix I'm on hold ;-)

If we have DIRECTORIES_TO_CREATE how will we treat their permissions during creation?
Or maybe another corner case (which would make my patch invalid btw.), what if /var/tmp was symlink to /tmp on original system?

V.

[jsmeix]

@gozora @schlomo
have a look at https://github.com/rear/rear/pull/1457
what you think about it.
I think it is much more generic and cleaner now.

@gozora
regarding permissions or symlinks:
There is nothing new compared to how MOUNTPOINTS_TO_RESTORE
worked before - i.e. no support for permissions or symlinks
except permissions from the mountpoint_permissions file as before.
Support also such advanced stuff was a reason behind my
https://github.com/rear/rear/issues/1455#issuecomment-324917669
which we might implement in the future if really needed
(a.k.a. if users really request it) but for now I agree with
https://github.com/rear/rear/issues/1455#issuecomment-324921139
and I also would like to keep things simple (at least for now).

On the other hand:
Currently DIRECTORIES_TO_CREATE is not really documented
in default.conf (same as MOUNTPOINTS_TO_RESTORE was
not really documented in default.conf before).
I think I could easily enhance DIRECTORIES_TO_CREATE
to a bash array of entries of the form

DIRECTORIES_TO_CREATE=( '/var/tmp 1777 root root' 'sys 555 root root' ...)

where each array member is a string of words of the form

[/]path/to/directory permission userid groupid

which would support traditional permissions but no ACLs
and no symlinks.

[schlomo]

Sorry, I am against storing multi-word strings in a Bash array (wrong tool for the job). Let's discuss the details of that topic in #1457 instead to keep it all there.

[schlomo]

I am closing this issue because the problem described here is fixed and we continue to improve the related code in #1457

[jsmeix]

With https://github.com/rear/rear/pull/1459 merged
the whole "directories to be (re)-created" functionality
got overhauled and enhanced.

Of course if new issues appear now, I will fix them.