Zfs: Add support for Fedora 29

[cvoltz]

Fedora 29 was released 10/30/2018. An updated zfs-release RPM needs to be created and released (along with whatever backend work is required to setup the fc29 channel on the server). Let me know what I can do to help with this.

[mskarbek]

For now, you can easily create one yourself from sources. 0.7.11 works perfectly fine with 4.18.16 which is kernel version Fedra 29 was released with. Official packages for Fedora 29 most likely will be released when 0.7.12 will be ready. :)

[tonyhutter]

@cvoltz I just uploaded the 0.7.11 F29 RPMs, please give them a try.

[turkstra]

I just upgraded two of my systems to Fedora 29 using the newly released RPM/repo. Everything is working fine so far.

[cvoltz]

It didn't upgrade correctly.

I ran sudo dnf update -y . After the update, the zfs kernel module wasn't loaded and dkms status returned nothing. Previously, it had returned:

$ dkms status
spl, 0.7.11, 4.18.9-200.fc28.x86_64, x86_64: installed
zfs, 0.7.11, 4.18.9-200.fc28.x86_64, x86_64: installed

Looking through the log for the upgrade (upgrade.log), it had a bunch of complaints like this:

zavl.ko.xz:
 - Uninstallation
   - Deleting from: /lib/modules/4.18.16-300.fc29.x86_64/extra/
rmdir: failed to remove 'extra': Directory not empty
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

It is true that the extra directory wasn't empty (it has 144 files in various subdirectories) but that shouldn't be a problem.

I expected that the spl and zfs modules would have been installed in DKMS but they weren't even added. The RPMs were installed so I ran:

sudo dkms install spl/0.7.11
sudo dkms install zfs/0.7.111

They compiled and installed OK:

$ dkms status
spl, 0.7.11, 4.18.16-300.fc29.x86_64, x86_64: installed
zfs, 0.7.11, 4.18.16-300.fc29.x86_64, x86_64: installed

Then I was able to run modprobe zfs and get ZFS loaded OK.

On my CentOS systems, I have always been able to just run the update and the modules would get updated via DKMS and reloaded. With Fedora 29, the upgrade didn't even add the spl and zfs modules to DKMS. This seems like a bug.

As a test, I ran sudo dnf remove --noautoremove libnvpair1 libuutil1 libzfs2 libzpool2 spl spl-dkms zfs zfs-dkms to remove the packages. I then ran sudo dnf install zfs (see install.log) to install the Fedora 29 packages again. The DKMS modules were installed correctly and I was able to load the zfs kernel module OK. So, it looks like the problem only occurs when upgrading from the Fedora 28 packages to the Fedora 29.

I then removed the Fedora 29 packages, reinstalled the Fedora 28 packages, loaded the zfs kernel module, and then upgraded to the Fedora 29 packages. This reproduced the problem (DKMS modules not added). So, there is definitely a bug with upgrading.

[tonyhutter]

@cvoltz can you see if your dkms includes this fix?: https://github.com/dell/dkms/pull/60

[mskarbek]

Most likely it's a DKMS issue. During an upgrade from one release to another you are installing a new kernel and new modules in one transaction which, at least based on my experience, has proven to be highly unreliable. It's not a ZFS fault, it happens with any DKMS-based modules.

[cvoltz]

@mskarbek Since the problem occurred when I duplicated the problem by downgrading and then upgrading just the ZFS RPMs, using the same kernel (4.18.16-300.fc29.x86_64), I don't think your theory fits the facts in this case.

[turkstra]

@cvoltz, I've had similar issues in the past. I add an extra step after erasing the packages: go into /var/lib/dkms/spl and /var/lib/dkms/zfs and erased everything - all directories and symlinks - and then reinstall the packages. Not sure if that will make a difference here, but thought it might be worth mentioning. I will note that I actually didn't have to do this when upgrading to Fedora 29 this time...

[cvoltz]

@tonyhutter No, it does not. The output in the upgrade.log doesn't suggest that it took that branch incorrectly. During the installs, it showed:

 - Installation
   - Installing to /lib/modules/4.18.16-300.fc29.x86_64/extra/
Adding any weak-modules

depmod.....

DKMS: install completed.

From what I can see, it looks like it ran an uninstall after it had completed the install. Grepping through the log for _DKMS.*completed_ shows 11 _DKMS: uninstall completed._ messages followed by a _DKMS: install completed._ message, followed by 11 more uninstall messages, followed by another install message, followed by 2 more uninstall messages. Looking at the kernel which was being operated on, the multiple uninstalls are for the different kernel modules left in /lib/modules (even though the kernels are no longer installed). After the spl module had been removed from all of the kernels, it was then installed for current kernel. After the zfs module have been removed from all of the kernels, it was then installed for the current kernel. Then, the zfs module was removed from the current kernel and the spl module was removed from the current kernel. I think everything would have been OK if the last 2 uninstalls hadn't occurred. The question is why were they uninstalled after they had just been installed?

[cvoltz]

@turkstra I would consider requiring that the user to perform that step to be a bug. As it is, the upgrade.log shows:

Deleting module version: 0.7.11
completely from the DKMS tree.
------------------------------
Done.
Loading new zfs-0.7.11 DKMS files...

which I suspect is doing what you suggested. After the upgrade fails, the only thing in the /var/lib/dkms directory is the file dkms_dbversion.

[cvoltz]

I cleared out the old files in /lib/modules, removed the F29 packages, installed the F28 packages, and upgraded to the F29 packages again. Grepping through the output (reupdate.log) shows the problem pretty clearly:

$ egrep "^DKMS|Module|Kernel|Loading" reupdate.log
Module:  spl
Kernel:  4.18.16-300.fc29.x86_64 (x86_64)
DKMS: uninstall completed.
Loading new spl-0.7.11 DKMS files...
DKMS: install completed.
Module:  zfs
Kernel:  4.18.16-300.fc29.x86_64 (x86_64)
DKMS: uninstall completed.
Loading new zfs-0.7.11 DKMS files...
DKMS: install completed.
Module:  zfs
Kernel:  4.18.16-300.fc29.x86_64 (x86_64)
DKMS: uninstall completed.
Module:  spl
Kernel:  4.18.16-300.fc29.x86_64 (x86_64)
DKMS: uninstall completed.

Those last 2 uninstalls should not have occurred.

Here are the steps to reproduce:

# remove the existing packages
sudo dnf remove --noautoremove libnvpair1 libuutil1 libzfs2 libzpool2 spl spl-dkms zfs zfs-dkms

# install the F28 packages
sudo dnf install --disablerepo=zfs --enablerepo=zfs-28 zfs

# verify the F28 packages are installed
rpm --query libnvpair1 libuutil1 libzfs2 libzpool2 spl spl-dkms zfs zfs-dkms

# check if the DKMS modules are installed (they are)
dkms status

# upgrade to the F29 packages
sudo dnf upgrade |& tee upgrade.log

# verify the F29 packages are installed
rpm --query libnvpair1 libuutil1 libzfs2 libzpool2 spl spl-dkms zfs zfs-dkms

# check if the DKMS modules are installed (they aren't)
dkms status

These steps assume the file /etc/yum.repos.d/zfs.repo is from zfs-release or has at least:

[zfs]
name=ZFS on Linux for Fedora $releasever
baseurl=http://download.zfsonlinux.org/fedora/$releasever/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

and the file /etc/yum.repos.d/zfs-28.repo has:

[zfs-28]
name=ZFS on Linux for Fedora 28
baseurl=http://download.zfsonlinux.org/fedora/28/$basearch/
enabled=0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

[tonyhutter]

@cvoltz thanks for the reproducer. https://github.com/zfsonlinux/zfs/issues/6902 looks like a similar bug to your first comment (https://github.com/zfsonlinux/zfs/issues/8089#issuecomment-436407486), where there's both a kernel and a distro upgrade. However as you mention, it's only the distro that's being upgraded in your reproducer steps, so that may be a separate issue. My guess is that this bug is present in all distro upgrades, not just F29.

I'd recommend we continue the discussion in https://github.com/zfsonlinux/zfs/issues/6902, and close this bug, since the F29 RPMs are now available.

[cvoltz]

OK

[stewartadam]

We should also update the wiki, which is linked to on the ZFS on Linux homepage:
https://github.com/zfsonlinux/zfs/wiki/Fedora

Can we also get a zfs-testing 0.8.0rc1 build for Fedora 29? :)

[behlendorf]

Wiki updated. Good idea, I'll get the testing repo updated this week.