Nixpkgs: NixOS Tests broken in master and staging

Created on 24 Aug 2020 · 14Comments · Source: NixOS/nixpkgs

Describe the bug
Trying to build a nixosTests fails due to stage-1 failing during creation of extra-utils

builder for '/nix/store/z1cxv06bi8iihhcifik0aqllqdbxhwpi-extra-utils.drv' failed with exit code 1
cannot build derivation '/nix/store/h8zxyl8r36bl4av33l2fbq0wkjs775mi-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/dbl58yvryaxsd4qzmxc09z4npzl07m7s-initrd-linux-5.4.54.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qr9hy73gr3wwgk81ll2f063ry57idlk2-nixos-system-bbmaster-20.09.git.4cfd81acfa0M.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rylgj77220s22979zlasdyrmw6g4syvf-nixos-system-bbworker-20.09.git.4cfd81acfa0M.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/6hss4221vbsch61ka9pzpsw2sfhh3lcg-nixos-system-gitrepo-20.09.git.4cfd81acfa0M.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/5sfqhjd1jsbf117sjn70ik777b03lqhi-nixos-vm.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/kah87whvjpp1zxqgjikwwny0249idlgb-nixos-vm.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/sxlb9fh7fqylpzw96jlkg4kfd2nrwshm-nixos-vm.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/xghh70sibzsnw27amzjrli0mgrx39f9j-nixos-test-driver-buildbot.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4riphbxaqq2ql1kw0nfv8yhiiv08xjbx-vm-test-run-buildbot.drv': 1 dependencies couldn't be built

nix log:

Copying libs for executable /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/lib/ld-linux-x86-64.so.2
/nix/store/ljd94dg6vmw9lx4vkszvl109w9y164sg-find-libs/bin/find-libs: line 19: left: unbound variable
stripping (with command strip and flags -s) in /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/lib  /nix/store/75p0>
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/systemd-sysctl...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/dmsetup...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/systemd-udevd...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/v4l_id...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/busybox...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/udevadm...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/ata_id...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/mke2fs...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/lvm...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/e2fsck...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/blkid...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/cdrom_id...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/mdadm...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/fido_id...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/tune2fs...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/kmod...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/scsi_id...
patching /nix/store/75p0yg2l7lciimfjm6hdh67rl6lyy99i-extra-utils/bin/mdmon...
testing patched programs...
hello world
245

from my own trouble shooting, the erring block of code is this "initrd test block" https://github.com/NixOS/nixpkgs/blob/40d2968ebf3b2b4c547caaa776b6eb875e1e5f83/nixos/modules/system/boot/stage-1.nix#L194

To Reproduce

git checkout staging # or master
git pull
nix-build -A nixosTests.buildbot

Expected behavior
able to run tests successfully

Additional context
git bisect run nix-build -A nixosTests.buildbot pointed me to:

72c8ed038946850d345db13f30e259a368d7c3c5 is the first bad commit
commit 72c8ed038946850d345db13f30e259a368d7c3c5
Author: Florian Klink <[email protected]>
Date:   Mon Aug 19 02:08:46 2019 +0200

    systemd: build with cryptsetup and cryptsetup-generators

    There's a circular dependency to systemd via cryptsetup and lvm2
    (systemd -> cryptsetup -> lvm2 -> udev=systemd).

    However, cryptsetup only really needs the devmapper component shipped
    with lvm2. So build `pkgs.cryptsetup` with a lvm2 that doesn't come with
    udev.

 nixos/modules/system/boot/systemd.nix      | 2 +-
 pkgs/os-specific/linux/systemd/default.nix | 4 +++-
 pkgs/top-level/all-packages.nix            | 6 +++++-
 3 files changed, 9 insertions(+), 3 deletions(-)

but this could be misleading due to how staging git history works

Notify maintainers
cc @flokli

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.60, NixOS, 20.09.git.2710fed815aM (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(archlayperson): `"home-manager"`
 - channels(root): `"nixos-20.09pre239318.c59ea8b8a0e"`
 - channels(jon): `"home-manager, nixos-20.09pre239318.c59ea8b8a0e"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: nixosTests
# a list of nixos modules affected by the problem
module:

bug channel blocker module system nixos testing

Source

jonringer

All 14 comments

reverting 72c8ed038946850d345db13f30e259a368d7c3c5 does allow me to run tests again cc @flokli

jonringer on 24 Aug 2020

👍1

I am affected by this too. My configuration.nix does not build because of this.

dasj19 on 25 Aug 2020

IIRC, the latest breakage was due to some meson bump which failed to properly update some references in the binaries rpath, thus introducing unwanted dependencies.

This should however be fixed in staging, I'm not sure what's popping up now here. Will take a look today.

flokli on 25 Aug 2020

It's a bit tricky that the extra-utils failure is hidden sometimes and the build succeeds (e.g. on my sandboxed NixOS).

vcunat on 25 Aug 2020

The error shown in the log file (different hashes perhaps):

/nix/store/fpb6drmpkmbkk363c24nv5gh9jjn2my2-extra-utils/bin/dmsetup: symbol lookup error: /nix/store/fpb6drmpkmbkk363c24nv5gh9jjn2my2-extra-utils/bin/dmsetup: undefined symbol: dm_udev_create_cookie, version Base

vcunat on 25 Aug 2020

```$ /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup
/nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup: error while loading shared libraries: libblkid.so.1: cannot open shared object file: No such file or directory

$ ldd /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup
linux-vdso.so.1 (0x00007ffdc4d9a000)
libdevmapper.so.1.02 => /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/lib/libdevmapper.so.1.02 (0x00007f3a6cf4f000)
libc.so.6 => /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/lib/libc.so.6 (0x00007f3a6cd90000)
libblkid.so.1 => not found
libm.so.6 => /nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30/lib/libm.so.6 (0x00007f3a6cc50000)
libpthread.so.0 => /nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30/lib/libpthread.so.0 (0x00007f3a6cc2f000)
/nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/lib/ld-linux-x86-64.so.2 => /nix/store/bqbg6hb2jsl3kvf6jgmgfdqy06fpjrrn-glibc-2.30/lib64/ld-linux-x86-64.so.2 (0x00007f3a6cfd6000)

FRidh on 25 Aug 2020

I'm not getting that problem.

NixOS machine (it succeeds):

patching /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/bin/tune2fs...
testing patched programs...
hello world
245
        linux-vdso.so.1 (0x00007ffff7fd2000)
        libdevmapper.so.1.02 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libdevmapper.so.1.02 (0x00007ffff7f46000)
        libc.so.6 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libc.so.6 (0x00007ffff7d87000)
        libudev.so.1 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libudev.so.1 (0x00007ffff7d55000)
        libblkid.so.1 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libblkid.so.1 (0x00007ffff7d03000)
        libm.so.6 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libm.so.6 (0x00007ffff7bc2000)
        libpthread.so.0 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libpthread.so.0 (0x00007ffff7ba1000)
        /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/ld-linux-x86-64.so.2 => /nix/store/2pi6zgkwnr3zdslvlv16nixpzvbyjx1n-glibc-2.31/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fd3000)
        librt.so.1 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/librt.so.1 (0x00007ffff7b95000)
/dev/mapper: mkdir failed: Permission denied
Failure to communicate with kernel device-mapper driver.
Incompatible libdevmapper 1.02.173 (2020-08-09) and kernel driver (unknown version).
Command failed.
Library version:   1.02.173 (2020-08-09)
mdadm - v4.1 - 2018-10-01
[...]
/nix/store/6y1mxz5ljjz3w6dqnkwhxmyak0mlsxh7-extra-utils.drv

Non-NixOS machine (fails):

patching /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/bin/systemd-udevd...
testing patched programs...
hello world
245
        linux-vdso.so.1 (0x00007ffff7fd2000)
        libdevmapper.so.1.02 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libdevmapper.so.1.02 (0x00007ffff7f48000)
        libc.so.6 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libc.so.6 (0x00007ffff7d89000)
        libblkid.so.1 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libblkid.so.1 (0x00007ffff7d37000)
        libm.so.6 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libm.so.6 (0x00007ffff7bf6000)
        libpthread.so.0 => /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/libpthread.so.0 (0x00007ffff7bd5000)
        /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/lib/ld-linux-x86-64.so.2 => /nix/store/2pi6zgkwnr3zdslvlv16nixpzvbyjx1n-glibc-2.31/lib64/ld-linux-x86-64.so.2 (0x00007ffff7fd3000)
/nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/bin/dmsetup: symbol lookup error: /nix/store/5ggvfzxy42albgqm38qc2hx0b38cpk94-extra-utils/bin/dmsetup: undefined symbol: dm_udev_create_cookie, version Base
builder for '/nix/store/6y1mxz5ljjz3w6dqnkwhxmyak0mlsxh7-extra-utils.drv' failed with exit code 1

A difference I notice is that the "patching" (first shown line) happens in a different order.

vcunat on 25 Aug 2020

Yes, I have the failure as you describe (but on NixOS), but then when I checked the log (like you did) and ran dmsetup directly myself, I got the error and ldd then showed a library is missing. It did not show this during the extra-utils build.

$ nix-build nixos/release-small.nix -A nixos.tests.simple -K
...
$ cat /tmp/nix-build-extra-utils.drv-2/log
/nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup: symbol lookup error: /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup: undefined symbol: dm_udev_create_cookie, version Base
$ ldd /nix/store/0bkyaqz81g7q7fjgdnddszjif44hb088-extra-utils/bin/dmsetup
...

FRidh on 25 Aug 2020

To be clear, I locally modified the extra-utils build to invoke ldd.

I suspect that find-libs may need to be done in topological order. And the order is changed probably due to different FS for /nix/store – right now it works for me on btrfs and fails on ext4, but so far I've only tried those two machines.

vcunat on 25 Aug 2020

That makes sense. Running ext4 here by the way.

FRidh on 25 Aug 2020

We have two differently configured lvm2 packages in the closure, so that's apparently the source of the confusion.

vcunat on 25 Aug 2020

👍1

Comments, please: #96290

vcunat on 25 Aug 2020

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/marketing-team-can-we-present-nix-nixos-better/6249/106