Zfs: ZFS-KMOD don't work after update from RHEL 7.3 to RHEL 7.4

[chassepa]

System information

Type | Version/Name
--- | ---
Distribution Name | RHEL
Distribution Version | 7.4
Linux Kernel | 3.10.0-693.1.1.el7.x86_64
Architecture | x86_64
ZFS Version | 0.7.1-1
SPL Version | 0.7.1-1

Describe the problem you're observing

ZFS was install using zfs-kmod repository on RHEL 7.3 with kernel version 3.10.0-514.26.2.el7.x86_64. After updating to release 7.4 with kernel version 3.10.0-693.1.1.el7.x86_64, ZFS module was not reinstalled. Reinstalling with zfs-kmod did not fix the problem. We had to switch to DKMS install.

Describe how to reproduce the problem

Install a system with RHEL 7.3. Install ZFS using zfs-kmod. Then update to RHEL7.4.

Include any warning/errors/backtraces from the system logs

[chassepa]

I think this is the wrong place for that kind of issues. I should have sent this message to the zfs-discuss list.

Paul-André

[behlendorf]

Packages for 7.4 will be added once the 3.10.0-693.1.1.el7.x86_64 kernel is available in the CentOS CR repository so we can build packages with mock.

[hoonetorg]

http://mirror.centos.org/centos/7.3.1611/cr/x86_64/Packages/kernel-3.10.0-693.1.1.el7.x86_64.rpm
that one ^^^ ;)

[dcrdev]

Please please update the kmods for the 693 kernel, the mainline kernel has a nasty bug in it with nvme storage that causes issues with zfs; it's fixed in 693 though.

[dmaziuk]

sorry, searching for "centos 7.4" didn't find "RHEL 7.4" for some reason ;)

[behlendorf]

0.7.1 packages have been posted for RHEL/CentOS 7.4. Updated zfs-release package:

http://download.zfsonlinux.org/epel/zfs-release.el7_4.noarch.rpm

[dmaziuk]

you rock!

Just a reminder to everyone: it'll be installed as zfs.repo.rpmnew

[chassepa]

Thank You Brian for the update.

Now I understand that the releases of ZFS for RedHat/CenOS depend on
CentOS release date and not on RedHat release date.
So next time Red Hat will release a new branch, I will wait for the
release of the same branch on CentOS before trying to update ZFS.

Thanks again.

Paul-André

[hubslave]

When updating to EL7.4, will it be necessary to uninstall zfs first and then install it again as when updating from EL 7.2 to EL7.3?

"When updating to RHEL/CentOS 7.3 the existing kmod packages will not work due to upstream kABI changes in the 3.10.0-514 kernel series. After upgrading to 7.3 users must uninstall ZFS and then reinstall it as described in the kABI-tracking kmod section. Compatible kmod packages will be installed from the CentoS 7.3 repository."

https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS

[behlendorf]

@hubslave yes. Thanks for pointing this out, I'll update the wiki to also reference EL7.4

[dmaziuk]

FWIW I did yum in http://download.zfsonlinux.org/epel/zfs-release.el7_4.noarch.rpm, renamed/edited zfs.repo.rpmnew, ran yum -y up && reboot and my /tank was there afterwards.

[hubslave]

@dmaziuk so you did not have to do uninstall zfs - update to EL7.4 - install zfs process? everything worked fine?

@behlendorf please is it also necessary to export all pools before update of EL and reinstall of zfs?

[behlendorf]

@hub the updated EL7.4 user space utilities should work correctly with the 7.3 kernel modules. You shouldn't need to export the pools until you're ready to reboot on to the new kernel and zfs kmods.

[dmaziuk]

@hubslave: yes, everything worked fine without the uninstall or pool export/import. FVO "fine" = 7.3 modules get uninstalled automagically -- you still can't boot into previous kernel and have zfs there. :(
.

[snehring]

@behlendorf would it be possible to get the 0.6.5.11 release added to this repo as well?

[behlendorf]

@snehring let me see what I can do. It seems reasonable to me to at least make it available.

[behlendorf]

0.6.5.11 packages added for CentOS/RHEL 7.4. Install with sudo yum install spl-0.6.5.11 zfs-0.6.5.11.

[snehring]

@behlendorf greatly appreciated, thanks.

[hubslave]

I'm asking just out of curiosity - but what is the point of using version 6.5.11 if 7.1 is already released?

[snehring]

In our specific case, an abundance of caution. 0.7.1 is a pretty big change and we're content to wait a few additional releases for any potential bugs to become known.

[hubslave]

@behlendorf I'm wondering, (I have no problem with uninstalling, updating to 7.4 and then installing again) but I'm just curious - I still have EL7.3 installed - would there be some problem if I added new ZFS repo for EL7.4 and did yum update and update ZFS together with EL packages when updating to EL7.4? What would happen if I did not uninstall ZFS before? I mean the old packages get overwritten by the new packages - so it is the same case as if I uninstalled it, did update to EL7.4 and then installed it again... Or not? Could there be some issues? 😕

[hubslave]

After the reboot, when all packages will be updated to EL7.4 and new kernel will become active, all the zfs packages should be compatible and working or not? :confused:

[Yevgeny1]

Distribution Name | CentOS
Distribution Version | 7.4
zfs-release.el7_4.noarch.rpm
Linux Kernel | 3.10.0-693.2.2.el7.x86_64

zfs-kmod and spl-kmod write modules to //3.10.0-693.1.1 path , even if
I not installed 3.10.0-693.1.1 kernel .

[ronbosworth]

Not sure if this is the best place for this information, but this was one of the top google hits when I went searching for what went wrong!

If you run Centos 7 and you recently updated using yum upgrade, you may find that after a reboot zfs no longer works:

~]# zfs list
The ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them.

Running modprobe does not help!

~]# /sbin/modprobe zfs
modprobe: FATAL: Module zfs not found.

You can fix this by uninstalling zfs, and following part of the official ZFS on Linux install guide

yum remove zfs zfs-kmod spl spl-kmod libzfs2 libnvpair1 libuutil1 libzpool2 zfs-release
yum install http://download.zfsonlinux.org/epel/zfs-release.el7_4.noarch.rpm
yum install zfs

ZFS should now work after another reboot or by running /sbin/modprobe zfs and importing your pools with zpool import.

I hope this helps other people.

[Yevgeny1]

"You can fix this by uninstalling zfs, and following part of the official ZFS on Linux install guide

yum remove zfs .....
yum install //// yum install zfs

ZFS should now work after another reboot or by running /sbin/modprobe zfs and importing your pools with zpool import.
I hope this helps other people."

Unfortunately, no.
zfs-kmod , when I try install it with this manual, installed non-correctly - it was installed to path for non-exist kernel version,
zfs-kmod and spl-kmod write modules to //3.10.0-693.1.1 path , even if
I not installed 3.10.0-693.1.1 kernel .

---

dkms works, but also has problems with OEL 7.4 (with 2 different kernels, same problem with version/path.)

[hubslave]

Today I did update - in my case zfs-kmod works without problems, I did not even need to uninstall it. I just updated zfs and my system to EL7.4 together in one update and after reboot everything worked fine out of the box. @behlendorf I just got an idea - I use few packages from elrepo.org and when there is new point version of EL7 (for example 7.3, 7.4...) they make its kernel a dependency for new kmod packages they create, so people can not accidentaly update kmod package to newer version if they dont have supported kernel, but when new kernel becomes available it is updated together with all new kmod packages. Maybe zfs-kmod in future releases could also work this way? Make kernel as a dependency and then there would be no need to create new zfs repositories for new point releases and also no need for uninstalling zfs and installing again after update...?

[dmaziuk]

IANA RPM guru but I think if you make kmod-zfs-1.2.3 depend on kernel > 3.10.500 then there will be a conflict when trying to install kmod-zfs-1.2.3 on kernel-3.10.499 -- i.e. it won't let you install 7.3 ZFS on top of 7.4 kernel. But it won't stop you from installing 7.4 kernel on top of 7.3 ZFS, which is what we're dealing with here.

If you pin to a specific kernel version, then you need to build a new RPM every time RHEL does a kernel rebuild.

I don't know exactly how ELRepo does their versioning, it may be worth contacting them and asking. I expect they struggled with this too.

In reality it's just another breakage in RedHat 7: 6 doesn't seem to have incompatible ABI changes in every minor release. No idea if this comes from upstream or some bright spark at RedHat comes up with a different set of custom patches on every build, but either way I wonder if their paying customers are screaming and taking their money to Canonical already.

[dmaziuk]

PS there is a related issue that RPM update removes the old version. Which in this case means if you ever boot into previous kernel, you won't have ZFS (I checked).

So if you want it really right, you have to build zfs-7.3 RPMs for kernels between 3.10.x and 3.10.y and zfs-7.4 RPMs for kernels between 3.10.y and whatever number they'll have for 7.5, and make the RPMs coexist...

[hubslave]

@dmaziuk You are right, if you make kmod-zfs-1.2.3 depend on kernel > 3.10.500 then there will be a conflict when trying to install kmod-zfs-1.2.3 on kernel-3.10.499 and that is what we want or not? This will prevent users installing new kmod packages on incompatible kernels. It wont make conflict if you try to install kernel 3.10.501 and higher... I mean I think that kmod packages are always tied to specific ABI arent they? So If there was new EL point release and ZFS would release new kmod-package dependent on new kernel, there would be no need of having new repository for ZFS every new EL point release, since this dependency would not allow users installing new kmod-zfs package on incompatible kernel, or in other words, the update would only run together with new kernel update. This means that after reboot everything should be up and running fine. No need for any complicated uninstall, update kernel and install again procedures or not? :)

[dmaziuk]

AFAIK the only real fix is switch to FreeBSD where ZFS is in the kernel in the first place.

[hubslave]

Well I'm pretty happy how ZFS is working on Linux I just wanted to point out that maybe it would be easier and less confusing for users to update it if kmod-zfs had similar dependencies as elrepo does for their kmod packages.