Zfs: mountpoint=legacy and fstab

[dmaziuk]

[root@batfish ~]# cat /etc/fstab
...
UUID=4f13ff87-ee44-4a92-a184-8b5578128ec1  /                           ext4  defaults        1 1
UUID=33a41311-cfdb-4f97-8562-f3911fa0caf0    swap                    swap defaults        0 0
UUID=850f14e2-2964-4ce0-af9e-4ae841445220 swap                    swap defaults        0 0
tank/extras                                                             /websites/extras  zfs     defaults        1 2
[root@batfish ~]# zfs get mountpoint tank/extras
NAME         PROPERTY    VALUE       SOURCE
tank/extras  mountpoint  legacy      local
[root@batfish ~]# mount /websites/extras
[root@batfish ~]# df
Filesystem      1K-blocks     Used  Available Use% Mounted on
...
tank/extras    2827994112 78874880 2749119232   3% /websites/extras

A reboot drops to "type control-D for maintenance" prompt, the last dmesg before that is

[ 10.472719] Adding 32767996k swap on /dev/sdb2.  Priority:-2 extents:1 across:32767996k FS
[   10.672016] SPL: using hostid 0x00000000

Now, interestingly enough

[root@batfish ~]# blkid /dev/sdb2
/dev/sdb2: UUID="850f14e2-2964-4ce0-af9e-4ae841445220" TYPE="swap"

-- the swap on /dev/sdb2 line in fstab is the one just above tank/extras.

This is on centos 7 with

[root@batfish ~]# rpm -q -a | grep zfs
zfs-dkms-0.6.5.1-1.el7.centos.noarch
zfs-release-1-2.el7.centos.noarch
zfs-0.6.5.1-1.el7.centos.x86_64
libzfs2-0.6.5.1-1.el7.centos.x86_64
[root@batfish ~]# uname -a
Linux batfish.bmrb.wisc.edu 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[dmaziuk]

PS. The problem I'm facing is the systems coming up with critical filesystems missing because dkms failed to build the modules.

[behlendorf]

@dmaziuk you might be interested in using the KMOD style packages for CentOS 7 this removes the need to rebuild ZFS via DKMS for every new kernel, see this comment for a longer description.

[dmaziuk]

I opened it as a separate issue because I want the system to not come up if zfs failed to mount for whatever reason. Dkms triggered it but there may be other reasons, like dead disks or ABI incompatibility.

[FransUrbo]

If this (https://github.com/zfsonlinux/zfs/issues/3812#issuecomment-142090689) is the actual problem, I fail to see what that have to do with having _legacy_ mounts in /etc/fstab.

[dmaziuk]

The real problem is the system coming up without critical filesystems mounted. /etc/fstab mounts is how I'm used to deal with that, if you have a viable alternative I'm all ears.

[FransUrbo]

There is no way, as far as I know, to make the boot up stop if a device isn't available by using /etc/fstab.

You can "fake" it by setting a fsck pass value > 0, but that's really cheating. AND you can't set that if it's a legacy zfs filesystem…

Besides, if you stop the boot process that early (importing or mounting ZFS pools/filesystems), you won't have something like sshd available and can't login to fix the issue. The ONLY time a boot up should fail that critically is if the root filesystem isn't available!

If you think that you really need/must have this, I would create my own checks and verification scripts, because that isn't "normal *NIX behavior".

[dmaziuk]

The only way it should fail so critically is when a critical filesystem is not available, / being _a_ critical filesystem but not _the only_ critical filesystem.

You can have custom startup scripts or you can have package managers, making both play nice together is a huge PITA and generally doesn't work. In the lennartware distros to e.g. prevent httpd from starting when /tank/website fails to mount requires modifying apache startup dependencies and your changes will be overwritten by the next yum up of course. If it's the webserver, "apache fails to start" = "the machine fails to start" i.e. it fails _that_ critically, you attach a console and start fixing it.

Cheating or not, setting fsck pass to not 0 is how we do it and that's what is not doable with zfs.

[FransUrbo]

prevent httpd from starting when /tank/website fails to mount requires modifying apache startup

If apache starts even though it's filesystem isn't available, that is an issue with apache, nothing else.
lack of sshd is a non-argument.

For you maybe, but not the majority.

And you don't need to modify each and every script. You write ONE script that makes sure everything is a-ok, put it at the relevant start level and you're good to go.

Any important server that doesn't run something a watchdog, isn't really important enough. I run watchdog, with /etc/watchdog.conf which lets me specify a whole range of things to guard. Such as "test-binary":

   test-binary = <testbin>
          Execute the given binary to do some user defined tests.

There are probably many more ways to solve this, these are two of the most obvious ones.

But since this is no longer about a problem with the fstab, could you please change the title of the issue to something like:

ZFS does not fail boot up if pool/filesystem is unavailable

I still don't think this is something that ZFS/ZoL should deal with, but that's my opinion. I consider this an administration issue, i.e., "your problem".

If you want to keep fighting for this, be my guest. An idea (or even better: actual code) on how this should be solved would be great to.

[dmaziuk]

Actually, the bug I'm reporting is that The Fine Manual says you can set mountpoint=legacy and mount a zfs filesystem via /etc/fstab and it doesn't work.

Until you rewrite the manual and call it a 'feature', it remains a bug.

[FransUrbo]

It DOES work just fine! But it can't mount something that doesn't exists...

[dmaziuk]

I just set up a clean centos 7 box, added kmod-zfs, created tank/test with mountpoint=legacy and

tank/test   /mnt  zfs  defaults  1 2

in /etc/fstab. From command line mount /mnt works just fine. Boot fails with Dependency Failed for File System Check on /tank/test.

So when you say "it does work", what exactly do you mean by "does" and "work"?

[FransUrbo]

tank/test /mnt zfs defaults 1 2
in /etc/fstab. From command line mount /mnt works just fine. Boot fails with Dependency Failed for File System Check on /tank/test.

I've never seen that error before, but if the actually error is 'File System Check', then that's obvious and we discussed that earlier - you can't have a fsck pass value greater than 0.

What 'Dependency Failed' means, I don't know...

[dmaziuk]

"Dependency Failed" is a lennartware thing, it's saying it's failed to reach its target "state". What's relevant is that with

tank/test   /mnt  zfs  defaults  0 0

it's still failing to get to multiuser runlevel. The only difference is now the error is "failed to mount /mnt" rather than "failed to fsck /mnt".

It's a good catch, though: I'm too used to typing 1 2 in fstab without thinking. Gotta remember to not do that anymore. Thanks.

[behlendorf]

@dmaziuk were you able to get to the root cause of this issue. Did updating the fstab to 0 0 resolve the issue?

[gmelikov]

Closed as stale. Feel free to reopen if you're able to reproduce it on fresh packages.

[cseufert]

Docker has created a bunch of images on my zfs root pool, with mountpoint=legacy, however, now my box won't boot, unless I zfs destroy all the offending mountpoints during the recover console, using Debian Stretch Backports version.

[cseufert]

Please see offending code for zfs initramfs script

        if [ "$mountpoint" = "legacy" -o "$mountpoint" = "none" ]; then
                # Can't use the mountpoint property. Might be one of our
                # clones. Check the 'org.zol:mountpoint' property set in
                # clone_snap() if that's usable.
                mountpoint=$(get_fs_value "$fs" org.zol:mountpoint)
                if [ "$mountpoint" = "legacy" -o \
                    "$mountpoint" = "none" -o \
                    "$mountpoint" = "-" ]
                then
                        if [ "$fs" != "${ZFS_BOOTFS}" ]; then
                                # We don't have a proper mountpoint, this
                                # isn't the root fs. So extract the root fs
                                # value from the filesystem, and we should
                                # (hopefully!) have a mountpoint we can use.
                                mountpoint="${fs##$ZFS_BOOTFS}"
                        else
                                # Last hail-mary: Hope 'rootmnt' is set!
                                mountpoint=""
                        fi
                fi

                if [ "$mountpoint" = "legacy" ]; 
# Should this actually happen here, I dont think we should be trying to automatically mount this
                        ZFS_CMD="mount -t zfs"
                else
                        # If it's not a legacy filesystem, it can only be a
                        # native one...
                        ZFS_CMD="mount -o zfsutil -t zfs"
                fi

To workaround this issue i have simply inserted the following code before the block shown above:

        if [ "$mountpoint" = "legacy" ]; then
                echo "Skipping legacy mount - $fs"
                return 0
        fi