zfs: error while loading shared libraries: libnvpair.so.1 (0.7.1 Regression vs 0.7.0)

Created on 30 Aug 2017 · 16Comments · Source: openzfs/zfs

Describe the problem you're observing

I built custom kmod packages of SPL and ZFS 0.7.1 using the instructions available here. The ./configure and compilation steps did not result in any errors, and I was able to successfully install all resulting packages. However, when running the zfs or zpool commands, I receive the following error:

zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory

This library is packaged by libnvpair1 and is located at /lib64. When I compiled SPL and ZFS 0.7.0 following the exact same procedure, I did not receive any errors running the zfs or zpool commands. Thus, this appears to be a regression in 0.7.1

Describe how to reproduce the problem

Compile ZFS and SPL 0.7.1 using the instructions here. Then attempt to run the zfs or zpool commands.

Packaging

Source

jwittlincohen

Most helpful comment

Per @BloodBlight, this worked (but going to try autoreconf next time)
```
ln -s /usr/local/lib/nvpair.a /usr/lib/nvpair.a
ln -s /usr/local/lib/libnvpair.la /usr/lib/libnvpair.a
ln -s /usr/local/lib/libnvpair.so /usr/lib/libnvpair.so
ln -s /usr/local/lib/libnvpair.so.1 /usr/lib/libnvpair.so.1
ln -s /usr/local/lib/libnvpair.so.1.0.1 /usr/lib/libnvpair.so.1.0.1
ln -s /usr/local/lib/libuutil.a /usr/lib/libuutil.a
ln -s /usr/local/lib/libuutil.la /usr/lib/libuutil.la
ln -s /usr/local/lib/libuutil.so /usr/lib/libuutil.so
ln -s /usr/local/lib/libuutil.so.1 /usr/lib/libuutil.so.1
ln -s /usr/local/lib/libuutil.so.1.0.1 /usr/lib/libuutil.so.1.0.1
ln -s /usr/local/lib/libzfs.a /usr/lib/libzfs.a
ln -s /usr/local/lib/libzfs_core.a /usr/lib/libzfs_core.a
ln -s /usr/local/lib/libzfs_core.la /usr/lib/libzfs_core.la
ln -s /usr/local/lib/libzfs_core.so /usr/lib/libzfs_core.so
ln -s /usr/local/lib/libzfs_core.so.1 /usr/lib/libzfs_core.so.1
ln -s /usr/local/lib/libzfs_core.so.1.0.0 /usr/lib/libzfs_core.so.1.0.0
ln -s /usr/local/lib/libzfs.la /usr/lib/libzfs.la
ln -s /usr/local/lib/libzfs.so /usr/lib/libzfs.so
ln -s /usr/local/lib/libzfs.so.2 /usr/lib/libzfs.so.2
ln -s /usr/local/lib/libzfs.so.2.0.0 /usr/lib/libzfs.so.2.0.0

PaulZ-98 on 6 Mar 2018

👍4 😄1

All 16 comments

When I compiled SPL and ZFS 0.7.0 following the exact same procedure, I did not receive any errors running the zfs or zpool commands.

On this same system? Is this an upgrade from a previous version or a new installation?

Can you rebuild 0.7.0 on the same box, install it, and verify both zfs and zpool start working again?

loli10K on 30 Aug 2017

@loli10K Yes, this is on the same system. It's a test VM that I created for this purpose. I removed all existing SPL and ZFS packages, followed the instructions for building/installing custom kmod packages, and then attempted to run the zpool and zfs commands. I actually removed the 0.7.1 packages and rebuilt the 0.7.0 packages last night, using the same instructions, just to make sure I hadn't done anything differently to cause the 0.7.1 failure. The 0.7.0 packages worked as expected.

jwittlincohen on 31 Aug 2017

@jwittlincohen this is something that would probably trigger failures on the buildbot.

root@debian-9:/usr/src/zfs# dpkg -l '*zfs*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                           Version                      Architecture                 Description
+++-==============================================-============================-============================-==================================================================================================
ii  kmod-zfs-4.9.0-2-amd64                         0.7.1-1                      amd64                        zfs kernel module(s) for 4.9.0-2-amd64
ii  kmod-zfs-devel                                 0.7.1-1                      amd64                        zfs kernel module(s) devel common
ii  kmod-zfs-devel-4.9.0-2-amd64                   0.7.1-1                      amd64                        zfs kernel module(s) devel for 4.9.0-2-amd64
ii  libzfs2                                        0.7.1-1                      amd64                        Native ZFS filesystem library for Linux
ii  libzfs2-devel                                  0.7.1-1                      amd64                        Development headers
ii  zfs                                            0.7.1-1                      amd64                        Commands to control the kernel modules and libraries
ii  zfs-dracut                                     0.7.1-1                      amd64                        Dracut module
ii  zfs-initramfs                                  0.7.1-1                      amd64                        Initramfs module
ii  zfs-test                                       0.7.1-1                      amd64                        Test infrastructure
root@debian-9:/usr/src/zfs# zfs list
no datasets available
root@debian-9:/usr/src/zfs#

Can you post the output of readelf -d /path/to/zfs and ldconfig -v 2>/dev/null | grep '^/' from both the working system (0.7.0) and 0.7.1?

root@debian-9:/usr/src/zfs# readelf -d `which zfs`

Dynamic section at offset 0x18d50 contains 36 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libnvpair.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuutil.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libzpool.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs_core.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libudev.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/lib64]
 0x000000000000000c (INIT)               0x4b28
 0x000000000000000d (FINI)               0x11ec4
 0x0000000000000019 (INIT_ARRAY)         0x218ad0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x218ad8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x298
 0x0000000000000005 (STRTAB)             0x1930
 0x0000000000000006 (SYMTAB)             0x2f8
 0x000000000000000a (STRSZ)              3452 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x219000
 0x0000000000000002 (PLTRELSZ)           5232 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x36b8
 0x0000000000000007 (RELA)               0x28d8
 0x0000000000000008 (RELASZ)             3552 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x000000006ffffffe (VERNEED)            0x2888
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x26ac
 0x000000006ffffff9 (RELACOUNT)          136
 0x0000000000000000 (NULL)               0x0
root@debian-9:/usr/src/zfs# ldconfig -v 2>/dev/null | grep '^/'
/usr/lib/x86_64-linux-gnu/libfakeroot:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
/lib:
/usr/lib:
root@debian-9:/usr/src/zfs# ls -l /lib64
total 11672
lrwxrwxrwx 1 root root       32 Aug  1 13:46 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.24.so
lrwxrwxrwx 1 root root       18 Aug 31 06:27 libnvpair.so -> libnvpair.so.1.0.1
lrwxrwxrwx 1 root root       18 Aug 31 06:26 libnvpair.so.1 -> libnvpair.so.1.0.1
-rwxr-xr-x 1 root root   335600 Aug 31 06:25 libnvpair.so.1.0.1
lrwxrwxrwx 1 root root       17 Aug 31 06:27 libuutil.so -> libuutil.so.1.0.1
lrwxrwxrwx 1 root root       17 Aug 31 06:26 libuutil.so.1 -> libuutil.so.1.0.1
-rwxr-xr-x 1 root root   295280 Aug 31 06:25 libuutil.so.1.0.1
lrwxrwxrwx 1 root root       20 Aug 31 06:27 libzfs_core.so -> libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root       20 Aug 31 06:26 libzfs_core.so.1 -> libzfs_core.so.1.0.0
-rwxr-xr-x 1 root root    81192 Aug 31 06:25 libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root       15 Aug 31 06:27 libzfs.so -> libzfs.so.2.0.0
lrwxrwxrwx 1 root root       15 Aug 31 06:26 libzfs.so.2 -> libzfs.so.2.0.0
-rwxr-xr-x 1 root root  1217976 Aug 31 06:25 libzfs.so.2.0.0
lrwxrwxrwx 1 root root       17 Aug 31 06:27 libzpool.so -> libzpool.so.2.0.0
lrwxrwxrwx 1 root root       17 Aug 31 06:26 libzpool.so.2 -> libzpool.so.2.0.0
-rwxr-xr-x 1 root root 10013568 Aug 31 06:25 libzpool.so.2.0.0
root@debian-9:/usr/src/zfs#

loli10K on 31 Aug 2017

@kpande To address the possibility of a configuration issue, I created a brand new Debian Stretch VM. It is totally stock with just the GNOME desktop and the required build dependencies for ZFS/SPL. I then rebuilt 0.7.1 using the custom build instructions. I verified the SHA256 hashes of the SPL and ZFS source downloads as well. The problem persists.

@loli10K
I built and tested 0.7.1 first. I then removed all SPL and ZFS packages and built 0.7.0.

0.7.1 Results

root@debian-vm:~# dpkg -l '*zfs*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  kmod-zfs-4.9.0 0.7.1-1      amd64        zfs kernel module(s) for 4.9.0-3-
ii  kmod-zfs-devel 0.7.1-1      amd64        zfs kernel module(s) devel common
ii  kmod-zfs-devel 0.7.1-1      amd64        zfs kernel module(s) devel for 4.
ii  libzfs2        0.7.1-1      amd64        Native ZFS filesystem library for
ii  libzfs2-devel  0.7.1-1      amd64        Development headers
ii  zfs            0.7.1-1      amd64        Commands to control the kernel mo
ii  zfs-dracut     0.7.1-1      amd64        Dracut module
un  zfs-fuse       <none>       <none>       (no description available)
ii  zfs-initramfs  0.7.1-1      amd64        Initramfs module
ii  zfs-test       0.7.1-1      amd64        Test infrastructure

root@debian-vm:~# readelf -d /sbin/zfs 

Dynamic section at offset 0x18d10 contains 39 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libnvpair.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuutil.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libzpool.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs_core.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libblkid.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libudev.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x4b88
 0x000000000000000d (FINI)               0x12114
 0x0000000000000019 (INIT_ARRAY)         0x218a88
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x218a90
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x298
 0x0000000000000005 (STRTAB)             0x1948
 0x0000000000000006 (SYMTAB)             0x2f8
 0x000000000000000a (STRSZ)              3498 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x219000
 0x0000000000000002 (PLTRELSZ)           5232 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x3718
 0x0000000000000007 (RELA)               0x2920
 0x0000000000000008 (RELASZ)             3576 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x000000006ffffffe (VERNEED)            0x28d0
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x26f2
 0x000000006ffffff9 (RELACOUNT)          136
 0x0000000000000000 (NULL)               0x0

root@debian-vm:/home/jason/Downloads/zfs-0.7.1# ldconfig -v 2>/dev/null | grep '^/'
/usr/lib/x86_64-linux-gnu/libfakeroot:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
/lib:
/usr/lib:



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:/usr/src/zfs-0.7.1# ls -l /lib64/
total 10012
lrwxrwxrwx 1 root root      32 Jun 15 15:17 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.24.so
lrwxrwxrwx 1 root root      18 Aug 31 22:29 libnvpair.so -> libnvpair.so.1.0.1
lrwxrwxrwx 1 root root      18 Aug 31 22:28 libnvpair.so.1 -> libnvpair.so.1.0.1
-rwxr-xr-x 1 root root  327920 Aug 31 22:28 libnvpair.so.1.0.1
lrwxrwxrwx 1 root root      17 Aug 31 22:29 libuutil.so -> libuutil.so.1.0.1
lrwxrwxrwx 1 root root      17 Aug 31 22:29 libuutil.so.1 -> libuutil.so.1.0.1
-rwxr-xr-x 1 root root  275496 Aug 31 22:28 libuutil.so.1.0.1
lrwxrwxrwx 1 root root      20 Aug 31 22:29 libzfs_core.so -> libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root      20 Aug 31 22:29 libzfs_core.so.1 -> libzfs_core.so.1.0.0
-rwxr-xr-x 1 root root   77864 Aug 31 22:28 libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root      15 Aug 31 22:29 libzfs.so -> libzfs.so.2.0.0
lrwxrwxrwx 1 root root      15 Aug 31 22:29 libzfs.so.2 -> libzfs.so.2.0.0
-rwxr-xr-x 1 root root 1191936 Aug 31 22:28 libzfs.so.2.0.0
lrwxrwxrwx 1 root root      17 Aug 31 22:29 libzpool.so -> libzpool.so.2.0.0
lrwxrwxrwx 1 root root      17 Aug 31 22:29 libzpool.so.2 -> libzpool.so.2.0.0
-rwxr-xr-x 1 root root 8367488 Aug 31 22:28 libzpool.so.2.0.0



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:/home/jason/Downloads/zfs-0.7.1# zpool list
zpool: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory
root@debian-vm:/home/jason/Downloads/zfs-0.7.1# zfs list
zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory



md5-08d38c2fa40b9ef0c959ba794fe093da



root@debian-vm:~# dpkg -l '*zfs*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  kmod-zfs-4.9.0 0.7.0-1      amd64        zfs kernel module(s) for 4.9.0-3-
ii  kmod-zfs-devel 0.7.0-1      amd64        zfs kernel module(s) devel common
ii  kmod-zfs-devel 0.7.0-1      amd64        zfs kernel module(s) devel for 4.
ii  libzfs2        0.7.0-1      amd64        Native ZFS filesystem library for
ii  libzfs2-devel  0.7.0-1      amd64        Development headers
ii  zfs            0.7.0-1      amd64        Commands to control the kernel mo
ii  zfs-dracut     0.7.0-1      amd64        Dracut module
un  zfs-fuse       <none>       <none>       (no description available)
ii  zfs-initramfs  0.7.0-1      amd64        Initramfs module
ii  zfs-test       0.7.0-1      amd64        Test infrastructure



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:~# readelf -d /sbin/zfs 

Dynamic section at offset 0x18d30 contains 36 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libnvpair.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuutil.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libzpool.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libzfs_core.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libudev.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [/lib64]
 0x000000000000000c (INIT)               0x4b60
 0x000000000000000d (FINI)               0x120f4
 0x0000000000000019 (INIT_ARRAY)         0x218aa8
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x218ab0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x298
 0x0000000000000005 (STRTAB)             0x1948
 0x0000000000000006 (SYMTAB)             0x2f8
 0x000000000000000a (STRSZ)              3459 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x219000
 0x0000000000000002 (PLTRELSZ)           5232 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x36f0
 0x0000000000000007 (RELA)               0x28f8
 0x0000000000000008 (RELASZ)             3576 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffb (FLAGS_1)            Flags: PIE
 0x000000006ffffffe (VERNEED)            0x28a8
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x26cc
 0x000000006ffffff9 (RELACOUNT)          136
 0x0000000000000000 (NULL)               0x0



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:/usr/src/zfs-0.7.0# ldconfig -v 2>/dev/null | grep '^/'
/usr/lib/x86_64-linux-gnu/libfakeroot:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
/lib:
/usr/lib:



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:/usr/src/zfs-0.7.0# ls -l /lib64/
total 10016
lrwxrwxrwx 1 root root      32 Jun 15 15:17 ld-linux-x86-64.so.2 -> /lib/x86_64-linux-gnu/ld-2.24.so
lrwxrwxrwx 1 root root      18 Aug 31 22:57 libnvpair.so -> libnvpair.so.1.0.1
lrwxrwxrwx 1 root root      18 Aug 31 22:57 libnvpair.so.1 -> libnvpair.so.1.0.1
-rwxr-xr-x 1 root root  327920 Aug 31 22:56 libnvpair.so.1.0.1
lrwxrwxrwx 1 root root      17 Aug 31 22:57 libuutil.so -> libuutil.so.1.0.1
lrwxrwxrwx 1 root root      17 Aug 31 22:57 libuutil.so.1 -> libuutil.so.1.0.1
-rwxr-xr-x 1 root root  275496 Aug 31 22:56 libuutil.so.1.0.1
lrwxrwxrwx 1 root root      20 Aug 31 22:57 libzfs_core.so -> libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root      20 Aug 31 22:57 libzfs_core.so.1 -> libzfs_core.so.1.0.0
-rwxr-xr-x 1 root root   77864 Aug 31 22:56 libzfs_core.so.1.0.0
lrwxrwxrwx 1 root root      15 Aug 31 22:57 libzfs.so -> libzfs.so.2.0.0
lrwxrwxrwx 1 root root      15 Aug 31 22:57 libzfs.so.2 -> libzfs.so.2.0.0
-rwxr-xr-x 1 root root 1191952 Aug 31 22:56 libzfs.so.2.0.0
lrwxrwxrwx 1 root root      17 Aug 31 22:57 libzpool.so -> libzpool.so.2.0.0
lrwxrwxrwx 1 root root      17 Aug 31 22:57 libzpool.so.2 -> libzpool.so.2.0.0
-rwxr-xr-x 1 root root 8364904 Aug 31 22:56 libzpool.so.2.0.0



md5-2595eb8276d5620c5782b2ada313f2ee



root@debian-vm:/usr/src/zfs-0.7.0# zfs list
no datasets available
root@debian-vm:/usr/src/zfs-0.7.0# zpool list
no pools available

jwittlincohen on 1 Sep 2017

@jwittlincohen can you please share this test VM (the whole bzipped disk will suffice)?

loli10K on 1 Sep 2017

@loli10K Here you go: Debian Stretch VM

jwittlincohen on 2 Sep 2017

Somehow libtool thinks both "/lib64" and "/usr/lib64" are in ld search path ... which they don't really seem to be:

root@debian-vm:/home/jason/Downloads/zfs-0.7.1# egrep '^sys_lib_dlsearch_path_spec=' libtool 
sys_lib_dlsearch_path_spec="/lib64 /usr/lib64 /lib /usr/lib /usr/lib/x86_64-linux-gnu/libfakeroot /usr/local/lib /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu "
root@debian-vm:/home/jason/Downloads/zfs-0.7.1# ldconfig -v 2>/dev/null | grep '^/'
/usr/lib/x86_64-linux-gnu/libfakeroot:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
/lib:
/usr/lib:
root@debian-vm:/home/jason/Downloads/zfs-0.7.1#

Paths in sys_lib_dlsearch_path_spec will not be hardcoded into executables, so every binary is missing "/lib64" in its RPATH:

https://www.gnu.org/software/libtool/manual/html_node/libtool-script-contents.html#index-sys_005flib_005fdlsearch_005fpath_005fspec

Variable: sys_lib_dlsearch_path_spec

Expression to get the run-time system library search path. Directories that appear in this list are never hard-coded into executables.

loli10K on 2 Sep 2017

... and the reason is quite simple, actually: zfs-0.7.1 tarball contains a version of "config/libtool.m4" which probably comes from a different distro and, by default, includes "/lib64 /usr/lib64 /lib /usr/lib" in ld search path.

root@linux:/tmp# tar xvf zfs-0.7.1.tar.gz zfs-0.7.1/config/libtool.m4
zfs-0.7.1/config/libtool.m4
root@linux:/tmp# tar xvf zfs-0.7.0.tar.gz zfs-0.7.0/config/libtool.m4
zfs-0.7.0/config/libtool.m4
root@linux:/tmp# grep lib64 -r zfs-0.7.*
zfs-0.7.0/config/libtool.m4:  # 'ldconfig -N -X -v | grep ^/' on 64bit Fedora does not report /usr/lib64,
zfs-0.7.1/config/libtool.m4:  sys_lib_dlsearch_path_spec="/lib64 /usr/lib64 /lib /usr/lib"
root@linux:/tmp#

We probably should not include lt* files from a specific distro in release tarballs? I've always only used git to build ZFS (which is easier for development) but i had always thought release archives to be pristine tarballs from git?

loli10K on 2 Sep 2017

I confirm the same issue when setting up ZoL 0.7.1 on Ubuntu 16.04

I was able to work around the issue by symlinking /lib/x86_64 -> /lib64

rmarder on 2 Sep 2017

I see this as well, in the 0.7.2 tarballs. Thanks to rmarder, your symlink solution worked for me on Debian Stretch.

I screwed around with this for a while, thinking it was due to remnants of a previous version hanging around and being seen by ./configure.. I built it several times.. and it still couldn't find the library.

darrenfreeman on 7 Oct 2017

The tarballs are generated using Ubuntu 16.04 (libtool v2.4.6) and they are intended to be portable. Unfortunately, that's not always the case due to difference in package versions and bugs. It's possible generating the packages using a newer version of libtool would handle this case. Alternately, running autoreconf --force --install will regenerate the configure script using your version of the tools.

behlendorf on 10 Oct 2017

👍1

@behlendorf Thanks for the suggestion! It resolved the issue for me on Debian Testing.

Debian is actually using the same version of libtool - 2.4.6. If the tarballs are using libtool from Ubuntu 16.04, isn't it strange that @rmarder is seeing the same issue in Ubuntu 16.04? This issue doesn't impact 0.7.0; did something change as of 0.7.1?

jwittlincohen on 13 Oct 2017

I am also seeing this on 16.04 with 0.7.3.

UPDATE: Worked around by adding several sym links from /usr/local/lib to /usr/lib.

BloodBlight on 26 Oct 2017

👍1

I can confirm the same error on Debian Stretch 9.1 and zfs release 0.7.6

zeigerpuppy on 25 Feb 2018

PaulZ-98 on 6 Mar 2018

👍4 😄1

I continue to get the OP's error after building zfs from source for the several systems I run.

zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory

I just got back from this hour-or-so trip with the newly-released 0.8.0-RC5. The solution is the same as mentioned above by @PaulZ-98, @BloodBlight and no doubt by @behlendorf . However my density continues to increase, as each time this error occurs I forget, and execute more unnecessary steps to fix it than the previous time. Call it _Imploding Brightness_.

Sugesstion: Update the Source Build or Custom Packages Wiki to cover this issue so that it cannot be recreated by mass-generating, yet surface-area-reducing brains.

OS, Ubuntu 16.04 x64
Kernel: Mainline 4.14.XXX (the Mainline kernels are manually installed .deb builds. They do not come through apt repos)