Zfs: what can we do about "device-mapper: reload ioctl on osprober-linux-sdX" failures?

Created on 2 Jan 2020  路  4Comments  路  Source: openzfs/zfs

System information

Type | Version/Name
--- | ---
Distribution Name | Ubuntu
Distribution Version | 18.04.3 LTS
Linux Kernel | 4.15.0-70-generic
Architecture | x64
ZFS Version | 0.7.5
SPL Version | 0.7.5

Describe the problem you're observing

With zfsonlinux, every time osprober-linux-sdX runs during update-initramfs activity, it always attempts to reload ioctl on all drives, and if that drive is in use by zfs it fails and prints a lovely Command failed message.

device-mapper: reload ioctl on osprober-linux-sdX1  failed: Device or resource busy
Command failed

To the keen observer, we know this is to be ignored on ZFS owned drives, but many users would see this and be concerned assuming something went wrong.

What I find a bit unexplained too is that on my system it isn't complaining on _EVERY_ drive, only a few.

Ultimately this is probably a bug in initramfs or related package(s) for device mapper?... (if so might zfs maintainers know the best place to fork this issue to?)

Describe how to reproduce the problem

Use zfsonlinux, on Ubuntu 18.04 LTS, run an apt upgrade that involves kernel updates.

or, run apt autoremove that involves wiping older kernels.

Include any warning/errors/backtraces from the system logs

# 1)
apt upgrade -y

# 2)
apt autoremove -y

which can result in

# /usr/bin/apt autoremove -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-headers-4.15.0-64 linux-headers-4.15.0-64-generic linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic linux-image-4.15.0-64-generic
  linux-image-4.15.0-72-generic linux-modules-4.15.0-64-generic linux-modules-4.15.0-72-generic linux-modules-extra-4.15.0-64-generic
  linux-modules-extra-4.15.0-72-generic
0 upgraded, 0 newly installed, 10 to remove and 0 not upgraded.

...

Removing linux-headers-4.15.0-64-generic (4.15.0-64.73) ...
Removing linux-headers-4.15.0-64 (4.15.0-64.73) ...
Removing linux-headers-4.15.0-72-generic (4.15.0-72.81) ...
Removing linux-headers-4.15.0-72 (4.15.0-72.81) ...
Removing linux-modules-extra-4.15.0-64-generic (4.15.0-64.73) ...
Removing linux-image-4.15.0-64-generic (4.15.0-64.73) ...
/etc/kernel/prerm.d/dkms:
dkms: removing: igb 5.3.5.22 (4.15.0-64-generic) (x86_64)

-------- Uninstall Beginning --------
Module:  igb
Version: 5.3.5.22
Kernel:  4.15.0-64-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

igb.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.15.0-64-generic/
rmdir: failed to remove '': No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-4.15.0-64-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-74-generic
Found initrd image: /boot/initrd.img-4.15.0-74-generic
Found linux image: /boot/vmlinuz-4.15.0-73-generic
Found initrd image: /boot/initrd.img-4.15.0-73-generic
Found linux image: /boot/vmlinuz-4.15.0-72-generic
Found initrd image: /boot/initrd.img-4.15.0-72-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
device-mapper: reload ioctl on osprober-linux-sda1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdd1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdf1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdg1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdk1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdl1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdn1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdo1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdr1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdu1  failed: Device or resource busy
Command failed
done
Removing linux-modules-extra-4.15.0-72-generic (4.15.0-72.81) ...
Removing linux-image-4.15.0-72-generic (4.15.0-72.81) ...
/etc/kernel/prerm.d/dkms:
dkms: removing: igb 5.3.5.22 (4.15.0-72-generic) (x86_64)

-------- Uninstall Beginning --------
Module:  igb
Version: 5.3.5.22
Kernel:  4.15.0-72-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

igb.ko:
 - Uninstallation
   - Deleting from: /lib/modules/4.15.0-72-generic/
rmdir: failed to remove '': No such file or directory
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-4.15.0-72-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.15.0-74-generic
Found initrd image: /boot/initrd.img-4.15.0-74-generic
Found linux image: /boot/vmlinuz-4.15.0-73-generic
Found initrd image: /boot/initrd.img-4.15.0-73-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
File descriptor 10 (/var/lib/dpkg/triggers/linux-update-4.15.0-64-generic (deleted)) leaked on lvs invocation. Parent PID 13179: /bin/sh
device-mapper: reload ioctl on osprober-linux-sda1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdd1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdf1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdg1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdk1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdl1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdn1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdo1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdr1  failed: Device or resource busy
Command failed
device-mapper: reload ioctl on osprober-linux-sdu1  failed: Device or resource busy
Command failed
done
Removing linux-modules-4.15.0-64-generic (4.15.0-64.73) ...
Removing linux-modules-4.15.0-72-generic (4.15.0-72.81) ...


ll /dev/disk/by-id | grep -v part | awk '{print $(NF-2) , $NF}' | grep sd
ata-HGST_HUS726T6TALE6L4_V8HED8BR ../../sdp
ata-Hitachi_HUA723030ALA640_MK0351YVK4U8DA ../../sdc
ata-HP_SSD_M700_120GB_HBSA18502000858 ../../sdi
ata-HP_SSD_M700_120GB_HBSA18502001148 ../../sdh
ata-ST2000VN000-1H3164_W1H25JXM ../../sdl
ata-ST2000VN004-2E4164_W52480RK ../../sdk
ata-ST4000VN003-2CN166_ZDH0HJ6A ../../sdr
ata-ST4000VN003-2CN166_ZDH0HJAE ../../sdg
ata-ST6000NE0023-2EX110_ZAD8X6WF ../../sds
ata-WDC_WD10PURX-64D85Y0_WD-WCC4J2PVET6D ../../sde
ata-WDC_WD10PURX-64D85Y0_WD-WCC4J7LPF1LA ../../sdb
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M5HPFCUC ../../sdo
ata-WDC_WD20EFRX-68EUZN0_WD-WCC4M6CLZFJC ../../sdn
ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5YK6CRE ../../sdf
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K3TR6014 ../../sdj
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5DFPY0F ../../sdm
ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHF92N ../../sda
ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0HHFYTV ../../sdu
wwn-0x5000c5006a406dab ../../sdl
wwn-0x5000c5009390052f ../../sdg
wwn-0x5000c50093902a06 ../../sdr
wwn-0x5000c5009d407083 ../../sdk
wwn-0x5000c500b69927ac ../../sds
wwn-0x5000cca098d42e63 ../../sdp
wwn-0x5000cca234ec76dc ../../sdc
wwn-0x50014ee20be7b33a ../../sdb
wwn-0x50014ee20be7b666 ../../sde
wwn-0x50014ee260fcdf59 ../../sda
wwn-0x50014ee262158a17 ../../sdf
wwn-0x50014ee2637acce9 ../../sdo
wwn-0x50014ee2b65283a4 ../../sdu
wwn-0x50014ee2b80f9f5a ../../sdn
wwn-0x50014ee2babf854d ../../sdj
wwn-0x50014ee2babf944e ../../sdm

All of my drives are whole-disk owned by ZFS except sdb and sde

# sudo fdisk -l  | grep ZFS
/dev/sda1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdb4  148441088 1953523711 1805082624 860.7G Solaris /usr & Apple ZFS
/dev/sde4  148441088 1953523711 1805082624 860.7G Solaris /usr & Apple ZFS
/dev/sdg1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdf1        2048 5860515839 5860513792  2.7T Solaris /usr & Apple ZFS
/dev/sdj1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdk1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdl1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdp1         2048 11721027583 11721025536  5.5T Solaris /usr & Apple ZFS
/dev/sdm1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdn1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdo1        2048 3907012607 3907010560  1.8T Solaris /usr & Apple ZFS
/dev/sdr1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdu1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sds1         2048 11721027583 11721025536  5.5T Solaris /usr & Apple ZFS
/dev/sdc1        2048 5860515839 5860513792  2.7T Solaris /usr & Apple ZFS

Also sde and sdh are owned by mdadm

# cat /proc/mdstat  | grep sd
md60 : active raid1 sdi2[3] sdh2[2]
md16 : active raid1 sdh3[2] sdi3[3]
Defect Question
Source
picture fermulator
👍10 👀1

Most helpful comment

This issue surfaced again for me in a recent upgrade of an Ubuntu 20.04 system using the experimental ZFS support and ZSYS, which calls update-grub and thus os-prober quite frequently after every stateful apt operation..

It is then also indirectly related to more recent reports we find in the following, where other side effects occur.

It is not possible for common people to regard this warning as purely cosmetical.

Do we know if s/o is working on a PR to os-prober that mitigates the warning by introducing ZFS to it, as suggested?
What can we do to help moving this out of sight?

picture almereyda on 23 Jul 2020
👍2

All 4 comments

Not sure if its related, but I saw some random "spammy-ish" logs similar too; sort of a "bulk spam" that occurred in a big chunk (maybe I ran scsi host scan on this day)

[Wed Jan  1 15:22:00 2020] device-mapper: table: 253:0: linear: Device lookup failed
[Wed Jan  1 15:22:00 2020] device-mapper: ioctl: error adding target to table
[Wed Jan  1 15:22:01 2020] device-mapper: table: 253:0: linear: Device lookup failed
[Wed Jan  1 15:22:01 2020] device-mapper: ioctl: error adding target to table
[Wed Jan  1 15:22:03 2020] device-mapper: table: 253:0: linear: Device lookup failed
[Wed Jan  1 15:22:03 2020] device-mapper: ioctl: error adding target to table
...
(snip)
...
fermulator picture fermulator on 18 Jan 2020

Link to the launchpad os-prober issue. Similar to mdadm, when ZFS is using a device it opens it for exclusive access (O_EXCL). This may be what's causing the warning. If this is the case, then os-prober will need to be updated to handle this is the same way it's handled for mdadm devices.

behlendorf picture behlendorf on 21 Jan 2020
👍1

This issue surfaced again for me in a recent upgrade of an Ubuntu 20.04 system using the experimental ZFS support and ZSYS, which calls update-grub and thus os-prober quite frequently after every stateful apt operation..

It is then also indirectly related to more recent reports we find in the following, where other side effects occur.

It is not possible for common people to regard this warning as purely cosmetical.

Do we know if s/o is working on a PR to os-prober that mitigates the warning by introducing ZFS to it, as suggested?
What can we do to help moving this out of sight?

almereyda picture almereyda on 23 Jul 2020
👍2

The warning is still happening today 9.1.2020 with Ubuntu 20.04's kernel upgrade.
device-mapper: reload ioctl on osprober-linux-sda1 failed: Device or resource busy

allanregistos picture allanregistos on 1 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

RNCTX picture RNCTX  路  3Comments
Greek64 picture Greek64  路  3Comments
pcd1193182 picture pcd1193182  路  4Comments
dbavatar picture dbavatar  路  4Comments
mailinglists35 picture mailinglists35  路  4Comments