Zfs: Package for EL7.7/EL8

[Tualua]

Hi! First of all - thank you for all your work on ZFS!

Only one question - is there a package for EL7.7/EL8? I can't find it in official guide for RedHat/CentOS

[behlendorf]

Not currently, for the moment you'll need to build your own packages. We expect to add these packages once the CentOS build of EL7.7/EL8 is released.

[dmfreemon]

FYI: CentOS 7.7 has been released.

[pod-adr]

It would be nice to get the most recent v0.8.1 at CentOS 7.7 and CentOS 8.

[mskarbek]

CentOS 8 will be released on September 24.

[Pascal666]

What happens if one of my boxes that automatically updated to 7.7 reboots while still on zfs from the 7.6 kmod repo?

[fpehla]

Upgrading to from Centos 7.6 (Kernel 3.10.0-957.27.2.el7.x86_64) to Centos 7.7 (Kernel 3.10.0-1062.1.1.el7.x86_64) using 0.7.x DKMS repos ZFS continues to work without issues. All pools are imported and running after reboot.

[dmfreemon]

Anyone using the kABI-tracking kmods lose access to their data if they do a yum update. I understand there are nuances, various interpretations associated with the definitions of what a release means, various perspectives on what a "production" server is and how "production" servers should be managed, and pragmatic concerns, but forcing everyone to build zfs from source after doing a routine yum update is not a great situation.

[UweSauter]

You cannot blame anyone but yourself. If you (auto-)update your system without checking if the ZFS packages are available no one can help you.
All my systems that are running CentOS and ZFS have exclude = kernel* zfs* spl* kmod-spl* kmod-zfs* libzfs2* libnvpair1* libzpool2* libuutil1* centos-release* in /etc/yum/yum-cron.conf so that no auto-update will do a release or kernel upgrade.

[dmfreemon]

I understand that excludes can be set up as you describe, but my comment is about policies and practices. Most people, most of the time, should be encouraged to routinely and frequently do yum updates. And yum updates should be a no-big-deal automated process that "just works". It is not realistic to expect everyone to grep around to figure out the correct list of excludes _apriori_. For if you don't get that list of excludes exactly right before the next nightly yum update runs, you're hosed.

[Pascal666]

Thank you UweSauter. I had exclude=kernel* under updates in CentOS-Base.repo, but the new kernel was pulled from base. I had no idea exclude would work in /etc/yum/yum-cron.conf. That really needs to be documented in the sample.

[guppy]

7.7 is working fine for me with kmods from the 7.6 rpm.

[ryantummy]

The 7.6 kmod repo works on 7.7 but it would be nice to know if the 7.7 repo will be out soon or if we have to use the 7.6 repo. Will this repo be available soon?

http://download.zfsonlinux.org/epel/zfs-release.el7_7.noarch.rpm

[Pascal666]

I'm glad to hear 7.7 works fine w/ 7.6 kmod repo. I'll probably stay there for a while. 7.7 zfs repo will have zfs 0.8. I'm not sure it is stable enough for production yet. I'll wait for v0.8.2 minimum, probably v0.8.3.

[ChibaPet]

FWIW, the DKMS code for ZOL 0.8.1 builds and runs under CentOS Stream 8. Both kmod variants fail building like this, which I haven't much explored yet:

make[2]: Leaving directory '/home/mason/zfs/zfs-0.8.1/module'
make \
top_distdir="zfs-0.8.1" distdir="zfs-0.8.1" \
dist-hook
make[2]: Entering directory '/home/mason/zfs/zfs-0.8.1'
./scripts/make_gitrev.sh
make[2]: [Makefile:1444: gitrev] Error 128 (ignored)
cp ./include/zfs_gitrev.h zfs-0.8.1/include; \
sed -i 's/Release:[[:print:]]/Release: 1/' \
zfs-0.8.1/META
make[2]: Leaving directory '/home/mason/zfs/zfs-0.8.1'
test -n "" \
|| find "zfs-0.8.1" -type d ! -perm -755 \
-exec chmod u+rwx,go+rx {} \; -o \
! -type d ! -perm -444 -links 1 -exec chmod a+r {} \; -o \
! -type d ! -perm -400 -exec chmod a+r {} \; -o \
! -type d ! -perm -444 -exec /bin/sh /home/mason/zfs/zfs-0.8.1/config/install-sh -c -m a+r {} {} \; \
|| chmod -R a+r "zfs-0.8.1"
tardir=zfs-0.8.1 && tar --format=posix -chf - "$tardir" | GZIP=--best gzip -c >zfs-0.8.1.tar.gz
gzip: warning: GZIP environment variable is deprecated; use an alias or script
{ test ! -d "zfs-0.8.1" || { find "zfs-0.8.1" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -fr "zfs-0.8.1"; }; }
make[2]: Entering directory '/home/mason/zfs/zfs-0.8.1'
make[2]: Leaving directory '/home/mason/zfs/zfs-0.8.1'
error: line 15: Dependency tokens must begin with alpha-numeric, '_' or '/': BuildRequires: %kernel_module_package_buildreqs
make[1]: [Makefile:1218: srpm-common] Error 1
make[1]: Leaving directory '/home/mason/zfs/zfs-0.8.1'
make: * [Makefile:1170: srpm-kmod] Error 2

[ChibaPet]

For ZOL 0.8.1 prerequisites on CentOS 8 and CentOS Stream, the following packages have been renamed:

python36-setuptools -> python3-setuptools
python36-cffi -> python3-cffi

[dacianstremtan]

I can also attest that the 7.6 kmod works on 7.7 I already updated about 10 servers and no issues. I'm using root zfs on these servers also.

[dmfreemon]

First, I want to say that my comments should not be taken negatively. ZFS on linux is outstanding software. I have used it for a long time. I'd like to see it used more widely, and I wonder if routine maintenance problems like this might be holding it back some in that regard. Here I'm referring to the dkms approach, which was never very solid in terms of trouble-free kernel updates.

Have not yet had a chance to recreate my original problem on one of my VM servers (to provide more specific details about my original failed upgrade attempt), but I did not imagine it. 'yum upgrade'ing from CentOS 7.6 to 7.7 did break my zfs due to the the kABI-tracking modules not loading. I now suspect the weak-updates symlinks got messed up during my yum update, but I cannot confirm that at the moment.

In the meantime, I will note the following:

We have seen this same (or similar) problem before, for example:
ZFS-KMOD don't work after update from RHEL 7.3 to RHEL 7.4
https://github.com/zfsonlinux/zfs/issues/6541

In zfsonlinux's own documentation:

"When updating to a new RHEL/CentOS 7.x release the existing kmod packages will not work due to upstream kABI changes in the kernel. After upgrading to 7.x users must uninstall ZFS and then reinstall it as described in the kABI-tracking kmod section. Compatible kmod packages will be installed from the matching CentOS 7.x repository."

https://github.com/zfsonlinux/zfs/wiki/RHEL-and-CentOS

This indeed matches expectations, as elrepo says:

"The Kernel Application Binary Interface (kABI) is a set of in-kernel symbols used by drivers and other kernel modules. Each major and minor RHEL kernel release has a set of in-kernel symbols that are whitelisted. A kABI-tracking kmod package contains a kernel module that is compatible with a given kABI, that is, for a given major and minor release of the EL kernel."

http://elrepoproject.blogspot.com/2016/02/kabi-tracking-kmod-packages.html

---

Using ZFS's 7.6 kABI-tracking kmods on CentOS 7.7 results in the following:

cat centos-release
CentOS Linux release 7.7.1908 (Core)

uname -a
Linux server01 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

yum list installed|grep zfs-kmod
kmod-spl.x86_64 0.7.13-1.el7_6 @zfs-kmod
kmod-zfs.x86_64 0.7.13-1.el7_6 @zfs-kmod
libnvpair1.x86_64 0.7.13-1.el7_6 @zfs-kmod
libuutil1.x86_64 0.7.13-1.el7_6 @zfs-kmod
libzfs2.x86_64 0.7.13-1.el7_6 @zfs-kmod
libzpool2.x86_64 0.7.13-1.el7_6 @zfs-kmod
spl.x86_64 0.7.13-1.el7_6 @zfs-kmod
zfs.x86_64 0.7.13-1.el7_6 @zfs-kmod

ksc -k /lib/modules/3.10.0-1062.1.1.el7.x86_64/weak-updates/zfs/zfs/zfs.ko
cat ksc-result.txt |grep ^(
(EdonRFinal)
(EdonRHash)
(EdonRInit)
(EdonRUpdate)
(SHA2Final)
(SHA2Init)
(SHA2Update)
(Skein_512_Final)
(Skein_512_InitExt)
(Skein_512_Update)
(__check_object_size)
(__cv_broadcast)
(__cv_destroy)
(__cv_init)
(__cv_signal)
(__cv_timedwait_sig)
(__cv_wait)
(__cv_wait_io)
(__cv_wait_sig)
(__free_pages)
(__kernel_fpu_begin)
(__kernel_fpu_end)
(__kstat_create)
(__kstat_delete)
(__kstat_install)
(__kstat_set_raw_ops)
(__remove_inode_hash)
(__set_page_dirty_nobuffers)
(__test_set_page_writeback)
(__thread_create)
(__thread_exit)
(__x86_indirect_thunk_r11)
(__x86_indirect_thunk_r12)
(__x86_indirect_thunk_r13)
(__x86_indirect_thunk_r14)
(__x86_indirect_thunk_r15)
(__x86_indirect_thunk_r8)
(__x86_indirect_thunk_r9)
(__x86_indirect_thunk_rax)
(__x86_indirect_thunk_rbx)
(__x86_indirect_thunk_rcx)
(__x86_indirect_thunk_rdx)
(_ctype)
(_raw_qspin_lock)
(avl_add)
(avl_create)
(avl_destroy)
(avl_destroy_nodes)
(avl_find)
(avl_first)
(avl_insert)
(avl_insert_here)
(avl_last)
(avl_nearest)
(avl_numnodes)
(avl_remove)
(avl_walk)
(bdi_setup_and_register)
(blk_queue_io_opt)
(blk_queue_max_segment_size)
(blk_queue_physical_block_size)
(blk_register_region)
(blk_unregister_region)
(capable)
(check_disk_change)
(cmn_err)
(crfree)
(crgetfsgid)
(crgetfsuid)
(crgetgid)
(crgetgroups)
(crgetngroups)
(crgetruid)
(crgetuid)
(crhold)
(cv_timedwait_hires)
(cv_timedwait_sig_hires)
(d_add_ci)
(d_invalidate)
(d_path)
(d_splice_alias)
(dataset_namecheck)
(ddi_copyin)
(ddi_copyout)
(ddi_strtoull)
(deactivate_locked_super)
(deactivate_super)
(do_sync_read)
(do_sync_write)
(downgrade_write)
(dput)
(drop_nlink)
(filemap_write_and_wait_range)
(find_lock_page)
(fletcher_2_incremental_byteswap)
(fletcher_2_incremental_native)
(fletcher_2_native)
(fletcher_4_abd_ops)
(fletcher_4_incremental_byteswap)
(fletcher_4_incremental_native)
(fletcher_4_native_varsize)
(fletcher_init)
(fnvlist_add_boolean)
(fnvlist_add_int32)
(fnvlist_add_nvlist)
(fnvlist_add_nvlist_array)
(fnvlist_add_nvpair)
(fnvlist_add_string)
(fnvlist_add_uint64)
(fnvlist_add_uint64_array)
(fnvlist_alloc)
(fnvlist_dup)
(fnvlist_free)
(fnvlist_lookup_nvlist)
(fnvlist_lookup_nvpair)
(fnvlist_lookup_string)
(fnvlist_lookup_uint64)
(fnvlist_num_pairs)
(fnvlist_pack)
(fnvlist_pack_free)
(fnvlist_remove_nvpair)
(fnvlist_size)
(fnvpair_value_int32)
(fnvpair_value_nvlist)
(fnvpair_value_string)
(fnvpair_value_uint64)
(follow_down_one)
(from_kgid)
(from_kuid)
(generic_end_io_acct)
(generic_file_mmap)
(generic_readlink)
(generic_start_io_acct)
(generic_write_checks)
(get_disk)
(get_gendisk)
(getrawmonotonic64)
(groupmember)
(ida_destroy)
(ida_init)
(ida_simple_get)
(ida_simple_remove)
(ilookup)
(inc_nlink)
(inode_change_ok)
(inode_set_flags)
(insert_inode_locked)
(irq_fpu_usable)
(kern_path)
(kill_anon_super)
(kmem_asprintf)
(kmem_debugging)
(kmem_vasprintf)
(kobj_close_file)
(kobj_get_filesize)
(kobj_open_file)
(kobj_read_file)
(kstat_runq_enter)
(kstat_runq_exit)
(kstat_waitq_enter)
(kstat_waitq_exit)
(lockref_get)
(make_kgid)
(make_kuid)
(match_strdup)
(match_token)
(new_inode)
(ns_capable)
(nv_alloc_fini)
(nv_alloc_init)
(nv_fixed_ops)
(nvlist_add_boolean)
(nvlist_add_boolean_array)
(nvlist_add_boolean_value)
(nvlist_add_byte)
(nvlist_add_byte_array)
(nvlist_add_int16)
(nvlist_add_int16_array)
(nvlist_add_int32)
(nvlist_add_int32_array)
(nvlist_add_int64)
(nvlist_add_int64_array)
(nvlist_add_int8)
(nvlist_add_int8_array)
(nvlist_add_nvlist)
(nvlist_add_nvlist_array)
(nvlist_add_nvpair)
(nvlist_add_string)
(nvlist_add_string_array)
(nvlist_add_uint16)
(nvlist_add_uint16_array)
(nvlist_add_uint32)
(nvlist_add_uint32_array)
(nvlist_add_uint64)
(nvlist_add_uint64_array)
(nvlist_add_uint8)
(nvlist_add_uint8_array)
(nvlist_alloc)
(nvlist_dup)
(nvlist_empty)
(nvlist_exists)
(nvlist_free)
(nvlist_lookup_boolean_value)
(nvlist_lookup_byte_array)
(nvlist_lookup_int32)
(nvlist_lookup_nv_alloc)
(nvlist_lookup_nvlist)
(nvlist_lookup_nvlist_array)
(nvlist_lookup_nvpair)
(nvlist_lookup_string)
(nvlist_lookup_uint64)
(nvlist_lookup_uint64_array)
(nvlist_merge)
(nvlist_next_nvpair)
(nvlist_pack)
(nvlist_prev_nvpair)
(nvlist_remove)
(nvlist_remove_all)
(nvlist_remove_nvpair)
(nvlist_size)
(nvlist_unpack)
(nvlist_xalloc)
(nvpair_name)
(nvpair_type)
(nvpair_value_boolean_value)
(nvpair_value_byte)
(nvpair_value_hrtime)
(nvpair_value_int16)
(nvpair_value_int16_array)
(nvpair_value_int32)
(nvpair_value_int32_array)
(nvpair_value_int64)
(nvpair_value_int64_array)
(nvpair_value_int8)
(nvpair_value_int8_array)
(nvpair_value_nvlist)
(nvpair_value_nvlist_array)
(nvpair_value_string)
(nvpair_value_uint16)
(nvpair_value_uint16_array)
(nvpair_value_uint32)
(nvpair_value_uint32_array)
(nvpair_value_uint64)
(nvpair_value_uint64_array)
(nvpair_value_uint8)
(nvpair_value_uint8_array)
(oops_in_progress)
(p0)
(path_get)
(path_put)
(pool_namecheck)
(posix_acl_chmod)
(posix_acl_create)
(posix_acl_equiv_mode)
(posix_acl_from_xattr)
(posix_acl_to_xattr)
(posix_acl_valid)
(random_get_pseudo_bytes)
(read_cache_pages)
(redirty_page_for_writepage)
(register_filesystem)
(register_shrinker)
(rootdir)
(rwsem_tryupgrade)
(schedule_timeout_interruptible)
(schedule_timeout_uninterruptible)
(security_inode_init_security)
(set_anon_super)
(set_nlink)
(sg_alloc_table)
(sg_free_table)
(sget)
(shrink_dcache_sb)
(simple_dir_inode_operations)
(simple_dir_operations)
(spl_kmem_alloc)
(spl_kmem_cache_alloc)
(spl_kmem_cache_create)
(spl_kmem_cache_destroy)
(spl_kmem_cache_free)
(spl_kmem_cache_reap_now)
(spl_kmem_cache_set_move)
(spl_kmem_free)
(spl_kmem_zalloc)
(spl_panic)
(spl_vmem_alloc)
(spl_vmem_free)
(spl_vmem_zalloc)
(strcspn)
(strdup)
(strfree)
(strlcat)
(strpbrk)
(system_delay_taskq)
(system_taskq)
(taskq_cancel_id)
(taskq_create)
(taskq_destroy)
(taskq_dispatch)
(taskq_dispatch_delay)
(taskq_dispatch_ent)
(taskq_init_ent)
(taskq_member)
(taskq_wait)
(taskq_wait_id)
(taskq_wait_outstanding)
(timespec_trunc)
(totalram_pages)
(touch_atime)
(truncate_inode_pages_range)
(truncate_setsize)
(tsd_create)
(tsd_destroy)
(tsd_get)
(tsd_set)
(u8_strcmp)
(u8_textprep_str)
(u8_validate)
(uio_prefaultpages)
(uiocopy)
(uiomove)
(uioskip)
(unlock_new_inode)
(unregister_filesystem)
(usecs_to_jiffies)
(vcmn_err)
(vmalloc_base)
(vmem_size)
(vn_close)
(vn_fsync)
(vn_getattr)
(vn_getf)
(vn_mode_to_vtype)
(vn_open)
(vn_openat)
(vn_rdwr)
(vn_releasef)
(vn_seek)
(vn_set_pwd)
(write_cache_pages)
(z_compress_level)
(z_uncompress)
(zfs_allocatable_devs)
(zfs_component_namecheck)
(zfs_deleg_verify_nvlist)
(zfs_deleg_whokey)
(zfs_name_to_prop)
(zfs_prop_default_numeric)
(zfs_prop_default_string)
(zfs_prop_get_type)
(zfs_prop_index_to_string)
(zfs_prop_inheritable)
(zfs_prop_init)
(zfs_prop_readonly)
(zfs_prop_setonce)
(zfs_prop_to_name)
(zfs_prop_user)
(zfs_prop_userquota)
(zfs_prop_valid_for_type)
(zfs_ratelimit)
(zfs_ratelimit_fini)
(zfs_ratelimit_init)
(zfs_spa_version_map)
(zfs_userquota_prop_prefixes)
(zfs_zpl_version_map)
(zio_arena)
(zone_get_hostid)
(zpool_get_rewind_policy)
(zpool_name_to_prop)
(zpool_prop_default_numeric)
(zpool_prop_feature)
(zpool_prop_get_type)
(zpool_prop_index_to_string)
(zpool_prop_init)
(zpool_prop_to_name)

ksc -k /lib/modules/3.10.0-1062.1.1.el7.x86_64/weak-updates/spl/spl/spl.ko
cat ksc-result.txt |grep ^(
(PDE_DATA)
(___ratelimit)
(__check_object_size)
(__kmalloc_node)
(__put_cred)
(__x86_indirect_thunk_r12)
(__x86_indirect_thunk_r13)
(__x86_indirect_thunk_r14)
(__x86_indirect_thunk_r8)
(__x86_indirect_thunk_rax)
(__x86_indirect_thunk_rdx)
(_ctype)
(_raw_qspin_lock)
(bit_wait)
(fget)
(getrawmonotonic64)
(inode_permission)
(io_schedule_timeout)
(kvasprintf)
(out_of_line_wait_on_bit)
(path_get)
(path_put)
(proc_doulongvec_minmax)
(rb_erase)
(rb_insert_color)
(register_shrinker)
(register_sysctl_table)
(schedule_hrtimeout_range)
(schedule_timeout_interruptible)
(set_user_nice)
(sigprocmask)
(sme_me_mask)
(usecs_to_jiffies)
(user_path_at)
(vfs_fsync)
(vfs_getattr)
(vfs_read)
(vfs_write)
(vmalloc_base)
(wake_up_bit)
(zlib_inflate)
(zlib_inflateEnd)
(zlib_inflateInit2)
(zlib_inflate_workspacesize)

Some of those symbols are obviously satisfied by other modules provided by the zfs software, but not all. This list is not comprehensive, as I am not listing all of the kernel modules provided by zfs.

To the extent the 7.6 kmods work on CentOS 7.7 may be in part just getting lucky.

The concern is that just because modules load and seems to work does that necessarily mean that everything is actually working correctly. How much testing has been done using 7.6 kmods on CentOS 7.7? Is the ZFS team endorsing running the 7.6 kmods on CentOS 7.7?

[llagendijk]

RHEL 8 seems to be changing the rpm macros. One needs to
yum install kernel-rpm-macros and manually add the build requires (stolen from the lustre spec file):
%if 0%{?rhel} >= 8
%if %{undefined kernel_module_package_buildreqs}
BuildRequires: redhat-rpm-config kernel-rpm-macros elfutils-libelf-devel kmod
%endif
%endif
I managed to compile the rpms, but cannot tst until I receive the new server in 2 weeks time

[tonyhutter]

FYI - 0.8.2 is out, which includes packages for Centos 7.7 (https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.2). We plan to build Centos 8 packages for 0.8.2, but we need buildbot to test it first, which is currently blocking on the Centos 8 AMIs getting released.

[kronenpj]

For what it's worth, I just tried a zfs upgrade from release-7_6 to release-7_7 (zfs 0.7.13 to 0.8.2) and it didn't go well. Dracut complained about not being able to mount root, even though zfs had mounted it in the right place.

[tsandmann]

I had the same problem, see #8913.

[pod-adr]

I have had no problems migrating from 7.6 to 7.7. How about CentOS 8? When could we expect it?

[guppy]

You can use the 7.7 RPM on CentOS 8 as long as you ignore a small error while installing the zfs-release RPM and have epel-release installed before doing "dnf install zfs". You must use DKMS also.

[dacianstremtan]

On Centos 8 I was able to manually compile the zfs-0.8.2. I recommend using the

make rpm

to build the required rpm packages and make a local repo to install it via yum. It will be much easier to upgrade it when needed or remove/downgrade the packages.

[tonyhutter]

@pod-adr we're waiting on the Centos 8 AMIs to be available on AWS. buildbot needs them to test 0.8.2 on Centos 8.

[JanusAlpha]

FYI been browsing AWS now and noticed Centos 8 minimalist AMIs. Go get'em boys (and gals). :)

[tonyhutter]

You'll see it here when it's officially released:
https://aws.amazon.com/marketplace/seller-profile/ref=srh_res_product_vendor?id=16cb8b03-256e-4dde-8f34-1b0f377efe89

It doesn't look like they've released one yet.

[7abbix]

It would be nice to get the most recent v0.8.1 at CentOS 7.7 and CentOS 8.

But CentOS 8 is already released some time ago.
Any info on when ZFS-kmod for EL8 is planned to be released?

[pod-adr]

I mean why do you have to make it via the official centos' ami? Can't it be just any Centos 8? God knows how long will they need for the official centos 8 ami and we want to use ZFS 0.8 on Centos 8 now. :) Or maybe a step-by-step user manual how to build it/sign it for UEFI under CentOS 8?

[7abbix]

People already almost getting to Mars and approaching to this by giant steps, but still no ZFS kmod on Centos 8. This is only packaging, right? So should be straightforward for community. Why so long? It is only small portion of software, which everyone is waiting for. A good practice is to prepare releases as soon as Enterprise Standard OS (=CentOS) major version comes out. Because it wasn't a sudden event, right? Everyone knew there will be Centos 8 soon and preview version was available for quite for a long time. Because of this ZFS technology is so slow into getting in Enterprises.

[mskarbek]

People already almost getting to Mars and approaching to this by giant steps, but still no ZFS kmod on Centos 8. This is only packaging, right? So should be straightforward for community. Why so long? It is only small portion of software, which everyone is waiting for. A good practice is to prepare releases as soon as Enterprise Standard OS (=CentOS) major version comes out. Because it wasn't a sudden event, right? Everyone knew there will be Centos 8 soon and preview version was available for quite for a long time. Because of this ZFS technology is so slow into getting in Enterprises.

Chill out, there was no publicly available beta for CentOS 8 release so there was no possibility to set up and upfront test the release process and as it was mentioned above there is still no official release of the AMI. It is not "only small portion", it is huge and in fact the most important one, you don't want to mess with storage software on the production. If it is so "straightforward" why don't you prepare those packages for yourself? If you want this done right and right now pay for it as a proper enterprise should for enterprise-class solutions then maybe one of the maintainers could spend enough time to prepare, test and do the release quickly enough for you.

[7abbix]

and as it was mentioned above there is still no official release of the AMI.

Excuse me, but what is AMI ? It is AWS compliant CentOS image ?

[kronenpj]

@7abbix Yes, that's Amazon's "Amazon Machine Image" virtual disk for running OS's on their platform.

[tonyhutter]

Our automated testing suite uses AMIs to test ZFS on different distributions. We need the Centos 8 AMIs to test out any Centos 8 packages we build. We wouldn't want to release packages that haven't been tested. The AMIs will show up here when they're released:
https://aws.amazon.com/marketplace/seller-profile?id=16cb8b03-256e-4dde-8f34-1b0f377efe89

[tonyhutter]

I've opened an issue requesting the Centos 8 AMIs:
https://bugs.centos.org/view.php?id=16614

[toddhpoole]

@tonyhutter, would now be the right time to start thinking about alternative build solutions for RHEL/CentOS packages?

Like you, we too have been waiting on CentOS, but there doesn't seem to be anyone at the wheel.

Bug reports remain open and untriaged, mailing list posts remain unanswered, and the project's Wiki page still contains information 9 months out of date.

It took the team 4.5 months to release CentOS 8, and despite a groundswell of folks volunteering to help on the mailing list, official responses ranged from "it's not a priority" to "I've been building CentOS releases for the last 17 years. I know how to do it. But thanks for the offer."

Given the speed with which RedHat continues to work on the RHEL 8.1 beta, we're now starting to wonder if RHEL 8.1 will get released before the CentOS 8.0 AMIs are even out...

[mskarbek]

@toddhpoole CentoOS 8 was a massive shift in the release process for the CentOS team, there are still things to do to automate and speed up the process. If you want to know with what issues they had to deal with then take a look at this year Floc presentation about building CentOS: https://www.youtube.com/watch?v=_iBcgZOL0to

[toddhpoole]

@mskarbek, no question. I think many would agree CentOS 8 was a huge undertaking, and the spike in mailing list activity back in September is a testament to that. But that doesn't mean the project gets a pass. (Not to imply that you were advocating it should.)

I recall Brian's talk, and of all the issues he correctly identified, he missed the most important ones: communication and community engagement. CentOS really has dropped the ball here lately.

From that Build Status page that was hardly ever updated, to all the folks - including former build engineers like myself - reaching out with unanswered offers to help, the project has not done well at communicating or engaging with the community.

These AMIs are another great example. ZFS on Linux is certainly not the only project blocked by these: others have also been asking how they can help move things along. Yet, despite their pings on Twitter, their comments on the CentOS Blog, their questions on the Forums, their bug reports on Mantis, and their posts on the mailing lists, there's been no update.

So given that whoever's responsible for these AMIs seems to be MIA, @tonyhutter, it might behoove us to start exploring other ways to build these packages.

[tonyhutter]

@toddhpoole yea, it's really frustrating. I talked to @behlendorf about the situation last week, and the new plan is to build the Centos 8 RPMs and try to test them against RHEL 8 AMIs. Hopefully RHEL 8 and Centos 8 are close enough that we can consider the test sufficient. At least then we're not blocking on the Centos 8 AMIs.

There's some issues building the 0.8.2 source RPMs under Centos 8 that I'm currently working though though...

[7abbix]

@toddhpoole Hopefully RHEL 8 and Centos 8 are close enough that we can consider the test sufficient. At least then we're not blocking on the Centos 8 AMIs.

They're not even close, but identical.
The Centos community only strips-off RHEL logos and keep everything original intact.

[mskarbek]

They're not even close, but identical.
The Centos community only strips-off RHEL logos and keep everything original intact.

It is not entirely true but "close enough". They are parts of the RHEL build environment that needs to be reverse-engineered by the CentOS team.

[7abbix]

CentOS - Community Enterprise OS.

It was developed by group of individuals interested to bring out an enterprise OS and they have taken RedHat and removed all the trademark related things logos and other things and recompiled it and brought it.
So it is an exact replica of RedHat and works just like RedHat.

You won't see there any RPM's suffixed with Centos.
So there is basically no difference. Everyone uses RPM packages suffixed as EL (EL8 in our case) and those known will work on both RedHat and Centos.

[7abbix]

The EL8 RPM for ZFSonLinux could be made long time ago. Of course if, there would be related test environment available.

[7abbix]

I can't remember, when I waited so hard on enterprise-important software not ready yet, but OS released long time ago. When RHEL8 was released, for example enterprise important software such as Net-SNMP 5.8 was already available.
As you can see, most enterprise important linux software communities manage to release RHEL compliant software really quick and are generally ready for new OS distro major release.
Do you know anything about how their work is organized ?
Why most enterprise important software communities are able to make RHEL compatible releases not waiting for availability of Centos AMI ?
One thing might be is that RHEL is based on specific Fedora version, which might be known in advance. They start developing a Fedora compatible release of specific version, which new RHEL will be based on. Then, when new RHEL comes out, it is up to packager to apply final tunings on .spec file.

[dominic-p]

Sorry to chime in here since I'm not really an expert. But, do you guys really want untested software released? Yes RHEL is very similar to CentoOS, but it's definitely not identical and testing on a different system than you run on doesn't make a lot of sense to me.

For a tool like ZFS stability and reliability are absolutely critical. If the testing infrastructure requires CentOS AMIs, then it makes a lot more sense to me to delay upgrading until they are available. You don't pick a distribution like CentOS if you want fast releases and access to the newest software. You pick it because you want stability and dependability. That only comes with rigorous testing.

Sorry again. It's not really my place, but if I was a ZoL maintainer this thread would really bum me out. So, I wanted to say thanks for all your hard work and I'm fine to wait until a proper testing environment is available. :)

[simlun]

@dominic-p don't be sorry. Thanks for chiming in.

Last I heard was that CentOS 7 is EOL 30 June 2024. So people stressing out about CentOS 8 just recently being released just need to chill out. They seriously have nothing to worry about if they really appreciate the <3 that is LTS releases.

And also, last I head was that ZoL devs aren't CentOS devs, so let _they_ do their awesome thing and let's let ZoL people do _their_ awesome thing.

I'm happily waiting for the ZoL devs to release some nice, stable and trustworthy RPM's when they are ready. Please ping us in this thread when it's done.

Best regards
Simon

[tonyhutter]

I will say that there's been more pain in building the Centos 8 RPMs than usual...
https://github.com/rpm-software-management/mock/pull/389

[tonyhutter]

...and this: https://github.com/zfsonlinux/zfs/issues/9546

[darkbasic]

If I understand correctly the problem is the lack of CentOS 8 AMIs, but there are RHE 8 AMIs available. Why don't you release packages for RHE 8 only and let people decide if they want to risk using them on CentOS as well? Then, once CentOS AMIs get available you start to officially support it as well: that way everybody would be happy I think.

[tonyhutter]

@darkbasic we're currently testing Centos 8 packages against RHEL8 AMIs, but we're hitting a kernel panic: https://github.com/zfsonlinux/zfs/issues/9546.

[tonyhutter]

To get around the #9546 kernel panic, I added the small blkg_tryget() modification to the kernel headers described here: https://github.com/zfsonlinux/zfs/issues/9546#issuecomment-549155776 before building the kmod packages. That allowed me finish testing the Centos 8 packages against RHEL 8.0. Note that the panic shouldn't be an issue in 0.8.3, since it will include https://github.com/zfsonlinux/zfs/commit/7ba964cc3f23e843fda0ac328997e25e54c79acb. It will be an issue though if you use the Centos 8 DKMS packages. If so, then you may want to apply the kernel header fix or the 7ba964cc3f23e843fda0ac328997e25e54c79acb patch before doing the dkms build.

Anyway, Centos 8.0 packages for 0.8.2 are now released:
http://download.zfsonlinux.org/epel/zfs-release.el8_0.noarch.rpm

[behlendorf]

It will be an issue though if you use the Centos 8 DKMS packages

Let me add that the only known device where we've encountered this issue is with the scsi_debug driver. Other rarely used block devices might have a similar issue, but the majority of block devices types are unaffected.

[sotiris-bos]

Why is there no zfs-dracut package in the kmod repo but there is one in the dkms?

Isn't it necessary for installing root on zfs?

[NoxHarmonium]

I had the same issue with zfs-dracut being missing from the kmod repo. I needed it to generate a initramfs that worked with zfs so I could use zfs as my root partition.

I downloaded the package from the dkms repo directly and installed it with rpm and it seemed to work.

curl -o /tmp/zfs-dracut-0.8.3-1.el8.noarch.rpm http://download.zfsonlinux.org/epel/8.1/kmod/x86_64/zfs-dracut-0.8.3-1.el8.noarch.rpm
rpm -ivh /tmp/zfs-dracut-0.8.3-1.el8.noarch.rpm

It didn't require any other dependencies or anything (AFAIK it is mostly shell scripts) so it seems safe to install outside of yum/dnf although I'll miss out on any updates.

I did some poking around and it doesn't seem like there is a corrosponding rpm in the kmod repo. This URL returns a 404:

http://download.zfsonlinux.org/epel/8.1/kmod/x86_64/zfs-dracut-0.8.3-1.el8.noarch.rpm

That URL returns a 404.

Also there it doesn't seem to be a reference to dracut in the kmod repo manifests.

$ curl -s http://download.zfsonlinux.org/epel/8.1/kmod/x86_64/repodata/ee6c2a82e2b41d49db22ac4039acd7aa7e6a6a11b344029afa70023e59dd6424-primary.xml.gz | gzip -d | grep dracut

$ curl http://download.zfsonlinux.org/epel/8.1/x86_64/repodata
  <name>zfs-dracut</name>
  <description>This package contains a dracut module used to construct an initramfs
  <location href="zfs-dracut-0.8.3-1.el8.noarch.rpm"/>
      <rpm:entry name="zfs-dracut" flags="EQ" epoch="0" ver="0.8.3" rel="1.el8"/>
      <rpm:entry name="dracut"/>

Although I might be doing something wrong!