The following three symbols are used by the SPL and are not part of the CentOS 7 kABI. For the epel kmod style packages to work properly for all CentOS 7 releases we need to eliminate our dependence on these symbols.
kernel(fput)
kernel(getrawmonotonic)
kernel(out_of_line_wait_on_bit)
behlendorf
All 17 comments
Official packages have been published for RHEL/CentOS 7.3. However, we should leave this issue open and investigate removing our dependency on any symbol not in the official whitelist to prevent this kind of issue in the future.
https://github.com/zfsonlinux/zfs/wiki/RHEL-&-CentOS#rhelcentos-73-kmod-packages
behlendorf
on 12 Dec 2016
For reference, further investigation of all symbols needed by ZFS on CentOS 7.5 shows that there are a total of 151 needed symbols which are not on the kABI whitelist. Many of the omitted symbols, such as unregister_filesystem(), are required by any filesystem.
behlendorf
on 18 Aug 2018
@behlendorf Does it means we will never have a zfs module which works on any RedHat/CentOS 7 kernel? Can I ask why we have such a module for RH/CentOS 6? Thanks.
shodanshok
on 9 Apr 2019
@shodanshok good question. The core issue is that kABI for RedHat/CentOS 7 doesn't include all of the symbols required by ZFS in its white list. This is part due to ZoL depending on a few more symbols than strictly required, and some unfortunate omissions in the kABI on RedHat's part.
This causes more of an issue for RedHat/CentOS 7 since it still receives a large number of backports from the upstream kernel which might change symbols used by ZoL which aren't part of the kABI white list. In which case the ZoL modules need to be rebuilt.
It's less of an issue for RedHat/CentOS 6 since the updates it's receiving at this point in its life cycle are very minimal. They're much less likely to change something ZoL depends on.
The good news is that for RedHat/CentOS 8 all the symbols required by ZoL have been included in the white list. So we should expect this to be much less of an issue.
behlendorf
on 11 Apr 2019
@shodanshok good question. The core issue is that kABI for RedHat/CentOS 7 doesn't include all of the symbols required by ZFS in its white list. This is part due to ZoL depending on a few more symbols than strictly required, and some unfortunate omissions in the kABI on RedHat's part.
This causes more of an issue for RedHat/CentOS 7 since it still receives a large number of backports from the upstream kernel which might change symbols used by ZoL which aren't part of the kABI white list. In which case the ZoL modules need to be rebuilt.
It's less of an issue for RedHat/CentOS 6 since the updates it's receiving at this point in its life cycle are very minimal. They're much less likely to change something ZoL depends on.
Ok, very clear. Thank for taking the time to explain.
The good news is that for RedHat/CentOS 8 all the symbols required by ZoL have been included in the white list. So we should expect this to be much less of an issue.
Excellent news, indeed!
shodanshok
on 11 Apr 2019
@behlendorf sorry to poke you again for a question about CentOS 7 packaging. Feel free to ignore the message or redirect me to the mailing list.
The current baseurl for CentOS 7 kmod packages is something as:
baseurl=http://download.zfsonlinux.org/epel/7.6/kmod/$basearch/
I wonder why we can't simply use the $releasever variable, as:
baseurl=http://download.zfsonlinux.org/epel/$releasever/kmod/$basearch/
This would enable a single zfs.repo file/package to serve the entire range of RHEL7 / CentOS 7 operating system. As things are today, we need a different zfs-release just to change/update the release version when OS is updated from a point release to the next one.
Am I missing something?
Thanks.
shodanshok
on 22 Sep 2019
@shodanshok it's my understanding that while $releasever is defined for Fedora it's not for CentOS 6/7. Although, I'd love to be wrong about this. It's entirely possible that functionality has since been added to CentOS. You can test this locally, if you update the baseurl in your configure to use $releasever does it expand to the correct location on your system?
behlendorf
on 24 Sep 2019
@behlendorf I really think $releasever is defined and usable for RHEL/CentOS. After all, the default, basic CentOS repository (CentOS-Base.repo) is define as follow:
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
As a direct test, I edited the zfs.repo file issuing, inside vim, the command %s/7.6/$releasever/g. The resulting file was:
...
[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/$releasever/kmod/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
...
A yum info zfs produces the correct results:
[root@localhost yum.repos.d]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base epel extras updates zfs-kmod
Cleaning up list of fastest mirrors
[root@localhost yum.repos.d]# yum info zfs
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink | 30 kB 00:00:01
* base: mirrors.prometeus.net
* epel: ftp.plusline.net
* extras: mirrors.prometeus.net
* updates: mirrors.prometeus.net
base | 3.6 kB 00:00:00
epel | 5.4 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
zfs-kmod | 2.9 kB 00:00:00
(1/8): base/7/x86_64/group_gz | 165 kB 00:00:00
(2/8): epel/x86_64/group_gz | 88 kB 00:00:01
(3/8): epel/x86_64/updateinfo | 1.0 MB 00:00:01
(4/8): base/7/x86_64/primary_db | 6.0 MB 00:00:02
(5/8): extras/7/x86_64/primary_db | 152 kB 00:00:00
(6/8): epel/x86_64/primary_db | 6.9 MB 00:00:01
(7/8): updates/7/x86_64/primary_db | 1.1 MB 00:00:01
(8/8): zfs-kmod/7/x86_64/primary_db | 246 kB 00:00:02
Installed Packages
Name : zfs
Arch : x86_64
Version : 0.7.13
Release : 1.el7_6
Size : 1.1 M
Repo : installed
From repo : zfs-kmod
Summary : Commands to control the kernel modules and libraries
URL : http://zfsonlinux.org/
License : CDDL
Description : This package contains the ZFS command line utilities.
Finally, yum-debug-dump:
%%%%YUM INFO
arch: ia32e
basearch: x86_64
releasever: 7
yum ver: 3.4.3
enabled plugins: fastestmirror
global excludes:
@shodanshok thanks for the info. Is $releasever dependent on the Centos you have installed? So if you have Centos 7 it will be 7.7, and if you have Centos 8 it will be 8.0?
tonyhutter
on 25 Sep 2019
Ah never mind, I think it will be "7" and "8" in those cases
tonyhutter
on 25 Sep 2019
Oh, that's right. There wasn't a mechanism to get the minor version which is why this was problematic.
behlendorf
on 25 Sep 2019
@tonyhutter Yes, $releasever would be "7" and "8" for CentOS 7.x and 8.x respectively.
@behlendorf The problem is related to correctly identify the minor RHEL/CentOS version? If so, you are right: there is no variable that I know to discover the minor version.
In this case, can I suggest a "zfs7_latest" repo which will use $releasever and kmod by default? This would decrease update complexity for everyone updating (and staying) to the latest RHEL/CentOS release, with the added convenience (read: faster install) of the kmod module vs the default dkms one of the other releases/repos.
shodanshok
on 25 Sep 2019
The good news is that for RedHat/CentOS 8 all the symbols required by ZoL have been included in the white list. So we should expect this to be much less of an issue.
@behlendorf Hi, tring to manually load the CentOS 8.0 zfs module in a newly installed CentOS 8.1, I get the following error: insmod: ERROR: could not insert module zfs.ko: Unknown symbol in module
It somewhat suprised me, as I (wrongly?) understood that RHEL/CentOS 8 has all the necessary symbols whitelisted.
Am I missing something?
Thanks.
shodanshok
on 15 Jan 2020
It somewhat surprised me, as I (wrongly?) understood that RHEL/CentOS 8 has all the necessary symbols whitelisted.
Unfortunately, I was slightly mistaken. As it turns out we were able to get the majority, but not all, of the symbols included in the whitelist. Would you mind checking dmesg and posting the symbol name which changed.
behlendorf
on 15 Jan 2020
@behlendorf Please disregard the unknow symbol error, as I did not load the modules in the proper order. Below you can find a more detailed (and hopefully correct) report:
[root@neutron 4.18.0-147.3.1.el8_1.x86_64]# modprobe zfs
modprobe: ERROR: could not insert 'zfs': Invalid argument
[root@neutron 4.18.0-147.3.1.el8_1.x86_64]# dmesg
[ 3593.839022] spl: disagrees about version of symbol filp_open
[ 3593.839026] spl: Unknown symbol filp_open (err -22)
[ 3593.839196] spl: disagrees about version of symbol fput
[ 3593.839197] spl: Unknown symbol fput (err -22)
[ 3593.839253] spl: disagrees about version of symbol kernel_read
[ 3593.839254] spl: Unknown symbol kernel_read (err -22)
[ 3593.839278] spl: disagrees about version of symbol filp_close
[ 3593.839279] spl: Unknown symbol filp_close (err -22)
[ 3593.839305] spl: disagrees about version of symbol kmalloc_caches
[ 3593.839306] spl: Unknown symbol kmalloc_caches (err -22)
[ 3593.839308] spl: disagrees about version of symbol kernel_write
[ 3593.839309] spl: Unknown symbol kernel_write (err -22)
[root@neutron weak-updates]# weak-modules --add-kernel --verbose
Module zfs.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: inode_init_once d_obtain_alias drop_nlink ilookup generic_file_llseek d_set_d_op set_nlink path_put super_setup_bdi_name clear_inode inc_nlink d_prune_aliases d_make_root path_get d_splice_alias current_time follow_down_one dput sget clear_nlink shrink_dcache_sb security_inode_init_security device_add_disk ida_simple_remove d_instantiate init_special_inode insert_inode_locked deactivate_super setattr_prepare ida_simple_get deactivate_locked_super unlock_new_inode touch_atime inode_owner_or_capable d_add_ci d_tmpfile iput new_inode __remove_inode_hash inode_set_flags ida_destroy set_anon_super d_invalidate generic_file_open igrab kill_anon_super kern_path
Module spl.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: filp_open fput kernel_read filp_close kmalloc_caches kernel_write
Module icp.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: spl_kmem_zalloc __cv_destroy spl_vmem_free spl_kmem_free __kstat_delete cmn_err strfree __cv_init spl_kmem_cache_alloc spl_vmem_alloc __kstat_create spl_kmem_cache_create __cv_wait __cv_broadcast taskq_dispatch taskq_wait __cv_signal spl_kmem_cache_destroy taskq_destroy kmem_asprintf spl_panic spl_kmem_cache_free spl_kmem_alloc taskq_create __kstat_install
Module znvpair.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: spl_kmem_free xdrmem_create ddi_strtol spl_vmem_alloc spl_panic spl_kmem_alloc
Module zavl.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: spl_panic
Module zcommon.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is not compatible with kernel 4.18.0-147.3.1.el8_1.x86_64 in symbols: spl_vmem_free spl_kmem_free __kstat_delete spl_vmem_alloc __kstat_create __kstat_set_raw_ops spl_panic spl_kmem_alloc __kstat_install
Module zunicode.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is compatible with kernel 4.18.0-147.3.1.el8_1.x86_64
Module zlua.ko from kernel 4.18.0-80.11.2.el8_0.x86_64 is compatible with kernel 4.18.0-147.3.1.el8_1.x86_64
@behlendorf can I do anything more to identify the missing symbols? Or the above is sufficient? Thanks.
shodanshok
on 23 Jan 2020
@shodanshok thanks, we got what we needed.
behlendorf
on 23 Jan 2020
Related issues
ltz3317
路
82Comments
wrouesnel
路
57Comments
runderwo
路
54Comments
mabod
路
53Comments
dmaziuk
路
52Comments