Zfs: Cannot set 'bootfs' property when 'dnodesize' is set to anything but legacy

[populationless]

System information

Type | Version/Name
--- | ---
Distribution Name | Arch Linux
Distribution Version | Rolling
Linux Kernel | 5.0.4
Architecture | x86_64
ZFS Version | c048ddaf332504765696a456ab572d6f041051a2
SPL Version | c048ddaf332504765696a456ab572d6f041051a2

Describe the problem you're observing

I'm currently running on VMware Workstation 15.0.3 vm. I setup Arch with native encryption like so:

$ zpool create -o ashift=12 -O acltype=posixacl -O canmount=off -O compression=lz4 -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/ -R /mnt $ rpool mirror /dev/disk/by-id/ata-disk-1-part2 /dev/disk/by-id/ata-disk-2-part2
$ zfs create -o canmount=off -o mountpoint=none -o encryption=on -o keyformat=passphrase rpool/enc
$ zfs create -o canmount=off -o mountpoint=none rpool/enc/ROOT
$ zfs create -o canmount=noauto -o mountpoint=/ rpool/enc/root/arch
...

The setup worked as expected, was able to boot using systemd-boot without issues.
I then wanted to play around with zedenv boot environments.

$ zedenv create arch-0
No bootfs has been set on zpool

Trying to set the bootfs property yields the following:

$ zpool set bootfs=rpool/enc/ROOT/arch
cannot set property for 'rpool': operation not supported on this type of pool

I'm not sure what's up. I googled around a bit but only came up with this, which I tried disabling compression but to no avail.

Describe how to reproduce the problem

Not sure atm, setting up another vm to test.

Include any warning/errors/backtraces from the system logs

AFAICT, there's nothing interesting in the log, but I'm kinda n00bish at this, so maybe point me in the right direction if more input is needed.

[populationless]

I figured it out. By creating a zpool with the dnodesize property set to anything but legacy, the bootfs property cannot be set.

$ zpool create -O dnodesize=legacy rpool /dev/disk/by-id/ata-disk-1
$ zfs create rpool/root
$ zpool set bootfs=rpool/root rpool
$ zpool get bootfs rpool
NAME     PROPERTY     VALUE     SOURCE
rpool     bootfs    rpool/root   local

$ zpool create -O dnodesize=auto|1k|2k|4k|8k|16k rpool /dev/disk/by-id/ata-disk-1
$ zfs create rpool/root
$ zpool set bootfs=rpool/root rpool
cannot set property for 'rpool': operation not supported on this type of pool
$ zpool get bootfs rpool
NAME     PROPERTY     VALUE     SOURCE
rpool     bootfs        -       default

I assume this is not intended behavior? If this is indeed a bug, what can I do to help investigate this further?

[grwilson]

Grub does not support the large_dnode feature so this is the expected behavior. The Arch wiki references the current ZFS features that grub supports.

[populationless]

I'm not using GRUB, I'm using systemd-boot (gummiboot) and can boot just fine with options zfs=rpool/enc/ROOT/arch rw. I was interested in the bootfs property for playing around with boot environments.

[grwilson]

I'm not familiar with that bootloader/bootmanager so I'm not sure if it can support all of the features that ZFS provides. The code assumes that the bootloader does not support this:

                        /*
                         * If this is a bootable dataset then
                         * we don't allow large (>512B) dnodes,
                         * because GRUB doesn't support them.
                         */
                        if (zfs_is_bootfs(dsname) &&
                            intval != ZFS_DNSIZE_LEGACY) {
                                return (SET_ERROR(EDOM));
                        }

                        if ((err = spa_open(dsname, &spa, FTAG)) != 0)
                                return (err);

                        if (!spa_feature_is_enabled(spa,
                            SPA_FEATURE_LARGE_DNODE)) {
                                spa_close(spa, FTAG);
                                return (SET_ERROR(ENOTSUP));
                        }

So if gummiboot is capable of handling this then there would need to be additional logic which checks if you're running on a feature-capable boot loader. Do you know if gummiboot support all ZFS pool features?

[populationless]

systemd-boot (formerly known as gummiboot) is a simple UEFI boot manager which is included in systemd. As far as I understand, it loads EFI executables from the ESP (EFI system partition) such as the Linux kernel EFISTUB. Since the kernel is doing all the heavy zfs lifting, it should support basically all availlable zfs features.

[grwilson]

Interesting. I will have to do more research on this (thanks for sharing). If you wanted to dig into this further then you could change the code above in zfs_ioctl.c to see if it would work. It sounds like the enhancement would be to detect if you're using an UEFI boot manager and then allow the setting of that feature/property.

[populationless]

Basically, a distributions kernel has to be built with "CONFIG_EFI_STUB" in order for it to be directly executable by the UEFI. Only caveat being it has to reside on the FAT32 formatted ESP. In theory, you wouldn't even need a classic 'bootloader' at all, you could just create a UEFI boot entry with the efibootmgr utility. See Debian wiki about EFIStub.
I will have a look at modifying that file and testing further. I'm fairly new to software development but this is exciting :)

[grwilson]

The challenge you will find is making a generic change safe for all. Since there are many ways to boot ZFS, a generic change needs to be able to safe guard against bricking a system while still allowing systems like yours to have full functionality. Thanks for looking into this.

[gmelikov]

Looks like it's about going away from grub. In this case we may remove grub-related restrictions from code.

[bunder2015]

just a side note, the "use the 8mb partition for efi" method only works if your kernel fits in that 8mb.

https://github.com/openzfs/openzfs/commit/7855d95b30

6277

[populationless]

Hm, not sure what you mean, but using the 8mb that remain at the end of a disk if you create a pool on a whole disk instead of a partition is too small for ESP. Currently, I create a 512MB ESP for a single linux distro (holds about 6 to 7 kernels) installed on root on zfs, or 1GB ESP or more if I want to have multiple distros and/or Windows dual boot on a single zfs root pool.

The openzfs commit looks like they implemente a -B switch to zpool create in order for it to create the ESP at pool creation time (if I unterstood correctly). Sounds interesting, maybe we should port this over?

[behlendorf]

We should definitely revisit porting openzfs/openzfs@7855d95 to Linux, it would be a nice feature. If I recall correctly the main reason it hasn't yet been done was setting up an environment to properly test it.

[rlaager]

~I gave that OpenZFS commit a quick read-through. The core assumptions about FAT32 size seem to be correct, which was one area I was initially concerned about.~

I'm less sure about this now. See: https://github.com/zfsonlinux/zfs/pull/7664#issuecomment-403234003

It seems that the minimum size is 257 MiB with 4K sectors. This works (but fails with 256 MiB):
truncate -s 257MiB test ; DEVICE=$(sudo losetup --show -b 4096 -f -P test) ; sudo mkdosfs -F 32 -s 1 "$DEVICE" ; sudo losetup -d "$DEVICE" ; rm test

[Echaleon]

Has there been any update to this? Just ran into this when trying to use zedenv.

[behlendorf]

@rlaager has opened https://github.com/zfsonlinux/zfs/pull/8627 which relaxes these restrictions.

[pcd1193182]

If we don't adopt that PR for whatever reason, we should also try to improve some of these error messages. Getting 'operation not supported on this type of pool' when the issue is that your dnodesize is set to auto instead of legacy is extremely confusing and non-intuitive.

[voidzero]

Ah. So that's why I couldn't set the bootfs property. Have been curious about this for a while. I'm using syslinux, by the way, which passes all of this on to the initrd anyway.

I guess a suitable workaround is to issue zfs inherit dnodesize pool, then zpool set bootfs and then set the dnodesize back to whatever it was before (auto in my case).