Nixpkgs: libgpg-error fails to cross-compile

[flokli]

Somewhere between the current nixpkgs-unstable channel and master libgpg-error cross-compilation broke.

It's a dependency of gpg, and systemd is usually compiled with a minimal gpg, to verify container image signatures - meaning this currently breaks NixOS cross. It's probably in the dependency closure of a lot of other stuff, so fixing it is probably better than building a systemd without a minimal gpg ;-)

https://github.com/NixOS/nixpkgs/pull/89219 seems unrelated - while it looks like this might have improve things, it was targeted to staging, is currently in staging-next, but not yet in master - so it broke before that, and still fails with it.

I'm currently bisecting this, but wanted to open a tracking issue, in case anyone else is running into this.

cc @NixOS/exotic-platform-maintainers

[flokli]

This has briefly been discussed in #nixos-dev yesterday: https://logs.nix.samueldr.com/nixos-dev/2020-06-17#3621161;

[erictapen]

I'd be not so sure that it wasn't #89219. Reverting 683004d092e6c4b5dc68d40684e8ecf65c87c1fc on top of nixos-unstable makes pkgsCross.aarch64-multiplatform.libgpgerror buildable for me.

According to their release notes they had cross fixes in both 1.37 and 1.38.

[flokli]

Hmm, my bisecting didn't get me too far:

git bisect start
# good: [9d0c3ffe6783d59b427d018e8341e0084737fde9] Merge pull request #90304 from r-ryantm/auto-update/binaryen
git bisect good 9d0c3ffe6783d59b427d018e8341e0084737fde9
# bad: [1357624fa9edff4add4c535c0606eb673f29966f] Merge pull request #90716 from doronbehar/update-zoom
git bisect bad 1357624fa9edff4add4c535c0606eb673f29966f
# bad: [b6108e021b557d4b9be5fb5927c12efa7f9ba419] Merge pull request #89327 from mweinelt/go-neb-module
git bisect bad b6108e021b557d4b9be5fb5927c12efa7f9ba419
# bad: [a038553c8b8cb108ecba63f77ac0f04917d29672] Merge #89901: dbus: 1.12.16 -> 1.12.18 (security)
git bisect bad a038553c8b8cb108ecba63f77ac0f04917d29672
# bad: [41dba08dd0ca5df86616f1857f8d01e715e7ceba] python: bugsnag: 3.6.0 -> 3.6.1
git bisect bad 41dba08dd0ca5df86616f1857f8d01e715e7ceba
# bad: [a337c44db65bdb7838bf1b1875631879b3a820c9] python3Minimal: disable optimizations
git bisect bad a337c44db65bdb7838bf1b1875631879b3a820c9
# skip: [af2dd0c798101dc112108859a3c427979d4802fc] Revert "Revert "Merge pull request #78910 from serokell/libarchive-zstd""
git bisect skip af2dd0c798101dc112108859a3c427979d4802fc
# skip: [7d5751412795e3a58cad82ac945eaa30377df746] meson: fix hash after incorrect merge
git bisect skip 7d5751412795e3a58cad82ac945eaa30377df746
# skip: [b76c308ecdc7a90179b49a55abca0058e481cb5b] Merge staging-next into staging
git bisect skip b76c308ecdc7a90179b49a55abca0058e481cb5b
# skip: [1c7d22e663ebbf76bf2a939994317a43dceb4c3a] ghostscript: 9.50 -> 9.52
git bisect skip 1c7d22e663ebbf76bf2a939994317a43dceb4c3a
# skip: [b7ff7465401257e9b0814bb68937a494c58de538] python3: now points to python38
git bisect skip b7ff7465401257e9b0814bb68937a494c58de538
# skip: [e761cfe50ac6a9204af703e2bb91e89ba521db02] libexif: 0.6.21 -> 0.6.22
git bisect skip e761cfe50ac6a9204af703e2bb91e89ba521db02
# skip: [bcf03e8cd2d1fb3a7d1dbbfee960dd47b36fc7bb] Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
git bisect skip bcf03e8cd2d1fb3a7d1dbbfee960dd47b36fc7bb
# skip: [0f2e569505597fc700049d06eeb54b3cf7656582] linux: CONFIG_MOUSE_ELAN_I2C_SMBUS=y
git bisect skip 0f2e569505597fc700049d06eeb54b3cf7656582
# skip: [6bfabdf1a22300731b63b2177caef9f232afab1a] openh264: 2.1.0 -> 2.1.1
git bisect skip 6bfabdf1a22300731b63b2177caef9f232afab1a
# skip: [55258867b922839d8349a2483ceeecf81703907c] pcre2: 10.34 -> 10.35
git bisect skip 55258867b922839d8349a2483ceeecf81703907c
# skip: [34aff7f6973743a5e6e672c01016c4280d471453] treewide: strip more things in cross
git bisect skip 34aff7f6973743a5e6e672c01016c4280d471453
# skip: [818fa358c83864c654f5e4c4d842915e7eb30237] Merge pull request #87954 from marsam/update-postgresql
git bisect skip 818fa358c83864c654f5e4c4d842915e7eb30237
# skip: [f8a8243bd3127ba9c6090d57f43d979389c1653c] cpython: Use --enable-optimizations, for a 16% speedup.
git bisect skip f8a8243bd3127ba9c6090d57f43d979389c1653c
# skip: [683004d092e6c4b5dc68d40684e8ecf65c87c1fc] libgpgerror: 1.36 -> 1.38
git bisect skip 683004d092e6c4b5dc68d40684e8ecf65c87c1fc
# skip: [68513786143e7698ad5dfab4e18b14b44b090822] postgresql_9_6: 9.6.17 -> 9.6.18
git bisect skip 68513786143e7698ad5dfab4e18b14b44b090822
# skip: [62ee86086c611e2209e11615cfc5fa96384ef468] libssh2: fix broken patch hash
git bisect skip 62ee86086c611e2209e11615cfc5fa96384ef468
# skip: [01377fc6ef7ab5092fd96a02a660c69805bbdc31] Merge pull request #87435 from matthewbauer/strip-more-in-cross
git bisect skip 01377fc6ef7ab5092fd96a02a660c69805bbdc31
# skip: [d89c58a012e4a8bc2d408bbc69a8d4bb1b265c44] perl: 5.30.2 -> 5.30.3
git bisect skip d89c58a012e4a8bc2d408bbc69a8d4bb1b265c44
# skip: [52c04b0347d60064f11ae8430e5a6c57a30ec831] cpython: Use autoreconfHook to rebuild configure script.
git bisect skip 52c04b0347d60064f11ae8430e5a6c57a30ec831
# skip: [c5669b0e475e7876eb96a3b3b4c95be049d4542c] Merge pull request #89441 from flokli/systemd-systemd.pc-system-user
git bisect skip c5669b0e475e7876eb96a3b3b4c95be049d4542c
# skip: [35d2d4d223d8993df12b70ffb312ead9f44cbf72] postgresql_10: 10.12 -> 10.13
git bisect skip 35d2d4d223d8993df12b70ffb312ead9f44cbf72
# skip: [b7b7da0521bbddf5725ff6259f51845de5f03311] python3Packages.cython: 0.29.14 -> 0.29.19
git bisect skip b7b7da0521bbddf5725ff6259f51845de5f03311
# good: [480c8d199166b2f8cd20e6e245d8a019329ec466] cpython: Optimize dynamic symbol tables, for a 6% speedup.
git bisect good 480c8d199166b2f8cd20e6e245d8a019329ec466
# skip: [073c37e06575f69be921840ca53c0a08a1793662] Merge pull request #89324 from doronbehar/libgme-gcc-ref
git bisect skip 073c37e06575f69be921840ca53c0a08a1793662
# skip: [3d2010311fd2280728dd404d4157086222a8c57e] gdb: 9.1 -> 9.2
git bisect skip 3d2010311fd2280728dd404d4157086222a8c57e
# skip: [3f08d642feb52ec08b09cff2ef668f3753a1803a] glibc: patch CVE-2020-1752
git bisect skip 3f08d642feb52ec08b09cff2ef668f3753a1803a
# skip: [ac0716aa6193d6ad72d255de38c9d0a6077ce8ee] iproute: 5.6.0 -> 5.7.0
git bisect skip ac0716aa6193d6ad72d255de38c9d0a6077ce8ee
# skip: [070c330f9b891513d61d2e3b8944bebc80850190] postgresql_11: 11.7 -> 11.8
git bisect skip 070c330f9b891513d61d2e3b8944bebc80850190
# skip: [da33b3e00a13bb41314e1c4cfdcc1c830a6a5fc6] Merge pull request #88266 from r-ryantm/auto-update/btrfs-progs
git bisect skip da33b3e00a13bb41314e1c4cfdcc1c830a6a5fc6
# skip: [c2039e1901aca58527d8c1a3206accac3b8482cb] mesa: make more things optional (#87438)
git bisect skip c2039e1901aca58527d8c1a3206accac3b8482cb
# skip: [3b1b4fde0cba48860c2db5dca0e39e6c3233a9c1] systemd: systemd.pc: fix systemdsystemunitpath and systemduserunitpath
git bisect skip 3b1b4fde0cba48860c2db5dca0e39e6c3233a9c1
# skip: [d1dcd310ddaf4dfcd192fb09e97c441836082cd8] jbig2dec: 0.17 -> 0.18
git bisect skip d1dcd310ddaf4dfcd192fb09e97c441836082cd8
# skip: [cf657dc584e296b629ac037aab583002b11166cf] libgme: Don't reference gcc-unwrapped
git bisect skip cf657dc584e296b629ac037aab583002b11166cf
# skip: [a2be64bf13bf23c69f5e67401bab96408cfd75cb] Merge pull request #84072 from gnprice/python-build
git bisect skip a2be64bf13bf23c69f5e67401bab96408cfd75cb
# skip: [51fd4ebc6ac7ebbe2ae1a785de955edbb816c9e8] Merge pull request #88842 from r-ryantm/auto-update/openh264
git bisect skip 51fd4ebc6ac7ebbe2ae1a785de955edbb816c9e8
# skip: [d2d070015b6daefc7dd5428b88cbfdca5de8d562] python: coverage: 4.5.4 -> 5.1
git bisect skip d2d070015b6daefc7dd5428b88cbfdca5de8d562
# skip: [490cce07a5ed11b0fba13b739a5246bf12991513] postgresql_12: 12.2 -> 12.3
git bisect skip 490cce07a5ed11b0fba13b739a5246bf12991513
# skip: [3ea0c1720086b247d2af54f8b99f142c48aae4a8] postgresql_9_5: 9.5.21 -> 9.5.22
git bisect skip 3ea0c1720086b247d2af54f8b99f142c48aae4a8
# skip: [54db65586179599a5a2b3dabff168523339f8bf6] btrfs-progs: 5.6 -> 5.6.1
git bisect skip 54db65586179599a5a2b3dabff168523339f8bf6
# only skipped commits left to test
# possible first bad commit: [a337c44db65bdb7838bf1b1875631879b3a820c9] python3Minimal: disable optimizations
# possible first bad commit: [bcf03e8cd2d1fb3a7d1dbbfee960dd47b36fc7bb] Revert "cpython: Optimize dynamic symbol tables, for a 6% speedup."
# possible first bad commit: [a2be64bf13bf23c69f5e67401bab96408cfd75cb] Merge pull request #84072 from gnprice/python-build
# possible first bad commit: [d2d070015b6daefc7dd5428b88cbfdca5de8d562] python: coverage: 4.5.4 -> 5.1
# possible first bad commit: [1c7d22e663ebbf76bf2a939994317a43dceb4c3a] ghostscript: 9.50 -> 9.52
# possible first bad commit: [d1dcd310ddaf4dfcd192fb09e97c441836082cd8] jbig2dec: 0.17 -> 0.18
# possible first bad commit: [e761cfe50ac6a9204af703e2bb91e89ba521db02] libexif: 0.6.21 -> 0.6.22
# possible first bad commit: [0f2e569505597fc700049d06eeb54b3cf7656582] linux: CONFIG_MOUSE_ELAN_I2C_SMBUS=y
# possible first bad commit: [55258867b922839d8349a2483ceeecf81703907c] pcre2: 10.34 -> 10.35
# possible first bad commit: [3d2010311fd2280728dd404d4157086222a8c57e] gdb: 9.1 -> 9.2
# possible first bad commit: [683004d092e6c4b5dc68d40684e8ecf65c87c1fc] libgpgerror: 1.36 -> 1.38
# possible first bad commit: [62ee86086c611e2209e11615cfc5fa96384ef468] libssh2: fix broken patch hash
# possible first bad commit: [d89c58a012e4a8bc2d408bbc69a8d4bb1b265c44] perl: 5.30.2 -> 5.30.3
# possible first bad commit: [b7b7da0521bbddf5725ff6259f51845de5f03311] python3Packages.cython: 0.29.14 -> 0.29.19
# possible first bad commit: [ac0716aa6193d6ad72d255de38c9d0a6077ce8ee] iproute: 5.6.0 -> 5.7.0
# possible first bad commit: [b7ff7465401257e9b0814bb68937a494c58de538] python3: now points to python38
# possible first bad commit: [7d5751412795e3a58cad82ac945eaa30377df746] meson: fix hash after incorrect merge
# possible first bad commit: [af2dd0c798101dc112108859a3c427979d4802fc] Revert "Revert "Merge pull request #78910 from serokell/libarchive-zstd""
# possible first bad commit: [b76c308ecdc7a90179b49a55abca0058e481cb5b] Merge staging-next into staging
# possible first bad commit: [073c37e06575f69be921840ca53c0a08a1793662] Merge pull request #89324 from doronbehar/libgme-gcc-ref
# possible first bad commit: [c5669b0e475e7876eb96a3b3b4c95be049d4542c] Merge pull request #89441 from flokli/systemd-systemd.pc-system-user
# possible first bad commit: [818fa358c83864c654f5e4c4d842915e7eb30237] Merge pull request #87954 from marsam/update-postgresql
# possible first bad commit: [da33b3e00a13bb41314e1c4cfdcc1c830a6a5fc6] Merge pull request #88266 from r-ryantm/auto-update/btrfs-progs
# possible first bad commit: [3b1b4fde0cba48860c2db5dca0e39e6c3233a9c1] systemd: systemd.pc: fix systemdsystemunitpath and systemduserunitpath
# possible first bad commit: [cf657dc584e296b629ac037aab583002b11166cf] libgme: Don't reference gcc-unwrapped
# possible first bad commit: [51fd4ebc6ac7ebbe2ae1a785de955edbb816c9e8] Merge pull request #88842 from r-ryantm/auto-update/openh264
# possible first bad commit: [3f08d642feb52ec08b09cff2ef668f3753a1803a] glibc: patch CVE-2020-1752
# possible first bad commit: [c2039e1901aca58527d8c1a3206accac3b8482cb] mesa: make more things optional (#87438)
# possible first bad commit: [01377fc6ef7ab5092fd96a02a660c69805bbdc31] Merge pull request #87435 from matthewbauer/strip-more-in-cross
# possible first bad commit: [6bfabdf1a22300731b63b2177caef9f232afab1a] openh264: 2.1.0 -> 2.1.1
# possible first bad commit: [54db65586179599a5a2b3dabff168523339f8bf6] btrfs-progs: 5.6 -> 5.6.1
# possible first bad commit: [490cce07a5ed11b0fba13b739a5246bf12991513] postgresql_12: 12.2 -> 12.3
# possible first bad commit: [070c330f9b891513d61d2e3b8944bebc80850190] postgresql_11: 11.7 -> 11.8
# possible first bad commit: [35d2d4d223d8993df12b70ffb312ead9f44cbf72] postgresql_10: 10.12 -> 10.13
# possible first bad commit: [68513786143e7698ad5dfab4e18b14b44b090822] postgresql_9_6: 9.6.17 -> 9.6.18
# possible first bad commit: [3ea0c1720086b247d2af54f8b99f142c48aae4a8] postgresql_9_5: 9.5.21 -> 9.5.22
# possible first bad commit: [34aff7f6973743a5e6e672c01016c4280d471453] treewide: strip more things in cross

I git bisect skiped all other build failures that weren't exactly the one as found here.

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
073c37e06575f69be921840ca53c0a08a1793662
b76c308ecdc7a90179b49a55abca0058e481cb5b
af2dd0c798101dc112108859a3c427979d4802fc
7d5751412795e3a58cad82ac945eaa30377df746
c5669b0e475e7876eb96a3b3b4c95be049d4542c
b7ff7465401257e9b0814bb68937a494c58de538
818fa358c83864c654f5e4c4d842915e7eb30237
ac0716aa6193d6ad72d255de38c9d0a6077ce8ee
b7b7da0521bbddf5725ff6259f51845de5f03311
d89c58a012e4a8bc2d408bbc69a8d4bb1b265c44
62ee86086c611e2209e11615cfc5fa96384ef468
683004d092e6c4b5dc68d40684e8ecf65c87c1fc
3d2010311fd2280728dd404d4157086222a8c57e
55258867b922839d8349a2483ceeecf81703907c
da33b3e00a13bb41314e1c4cfdcc1c830a6a5fc6
0f2e569505597fc700049d06eeb54b3cf7656582
3b1b4fde0cba48860c2db5dca0e39e6c3233a9c1
cf657dc584e296b629ac037aab583002b11166cf
51fd4ebc6ac7ebbe2ae1a785de955edbb816c9e8
e761cfe50ac6a9204af703e2bb91e89ba521db02
490cce07a5ed11b0fba13b739a5246bf12991513
d1dcd310ddaf4dfcd192fb09e97c441836082cd8
070c330f9b891513d61d2e3b8944bebc80850190
1c7d22e663ebbf76bf2a939994317a43dceb4c3a
3f08d642feb52ec08b09cff2ef668f3753a1803a
35d2d4d223d8993df12b70ffb312ead9f44cbf72
c2039e1901aca58527d8c1a3206accac3b8482cb
d2d070015b6daefc7dd5428b88cbfdca5de8d562
01377fc6ef7ab5092fd96a02a660c69805bbdc31
68513786143e7698ad5dfab4e18b14b44b090822
a2be64bf13bf23c69f5e67401bab96408cfd75cb
34aff7f6973743a5e6e672c01016c4280d471453
3ea0c1720086b247d2af54f8b99f142c48aae4a8
54db65586179599a5a2b3dabff168523339f8bf6
6bfabdf1a22300731b63b2177caef9f232afab1a
bcf03e8cd2d1fb3a7d1dbbfee960dd47b36fc7bb
a337c44db65bdb7838bf1b1875631879b3a820c9
We cannot bisect more!

@Ericson2314 did you get any further?

[cleverca22]

[clever@amd-nixos:~/apps/nixpkgs-gpg]$ git bisect log
git bisect start
# bad: [f7214facbf5668e2957fd763041b6a4ef990a10b] Merge pull request #74487 from oxalica/cargo-insta
git bisect bad f7214facbf5668e2957fd763041b6a4ef990a10b
# good: [45e01e508a792077c77067467e07bce757932fb6] Merge pull request #78716 from Ma27/bump-matrix-synapse
git bisect good 45e01e508a792077c77067467e07bce757932fb6
# good: [44e56c2503d0bfa60e200f9a0dfc1c12d0219b28] kmon: 1.0.0 -> 1.0.1
git bisect good 44e56c2503d0bfa60e200f9a0dfc1c12d0219b28
# good: [8a2688fbdbed0fd16f8f8644d3f4a6b9e993b41c] python3Packages.google-cloud-iot: 0.3.0 -> 0.3.1
git bisect good 8a2688fbdbed0fd16f8f8644d3f4a6b9e993b41c
# good: [0f114432d4a9399e0b225d5be1599c7ebc5e2772] Merge pull request #89193 from zowoq/minikube
git bisect good 0f114432d4a9399e0b225d5be1599c7ebc5e2772
# good: [9dd9bc7bcc2a54f7dc0cb23cef701439dfb9d0ca] linux: fix kernel config options
git bisect good 9dd9bc7bcc2a54f7dc0cb23cef701439dfb9d0ca
# bad: [6857bccf12e1b55b75e22342368feef22c78d7b9] Merge #89438: firefox updates (into staging-next)
git bisect bad 6857bccf12e1b55b75e22342368feef22c78d7b9
# good: [3437d3c64fd7578651d2695f9c43b4c972dfa9a5] Merge pull request #90154 from adisbladis/gimp-2_10_20
git bisect good 3437d3c64fd7578651d2695f9c43b4c972dfa9a5
# bad: [3b3d4f9dbef0e9c7258f1f66c55bdf07d6ef22b6] python: trimesh: 3.6.38 -> 3.6.43
git bisect bad 3b3d4f9dbef0e9c7258f1f66c55bdf07d6ef22b6
# bad: [5322686d537e30cd11c2bf135edb41578e570108] mailman: 3.3.0 -> 3.3.1
git bisect bad 5322686d537e30cd11c2bf135edb41578e570108

no skips in my log
but ive hit a stdenv rebuild, and need to head to bed

[erictapen]

Yeah bisecting stuff that went to staging is a PITA. Especially if it's cross stuff.

@flokli I recognised that 683004d092e6c4b5dc68d40684e8ecf65c87c1fc is part of your list of possible commits. I'm pretty sure this is an upstream problem.

[Ericson2314]

There are some fixes upstream since the release, trying them.

[cleverca22]

683004d092e6c4b5dc68d40684e8ecf65c87c1fc is the first bad commit
commit 683004d092e6c4b5dc68d40684e8ecf65c87c1fc
Author: zowoq <[email protected]>
Date:   Sun May 31 00:29:33 2020 +1000

    libgpgerror: 1.36 -> 1.38

bisect confirms that the problem did happen at a recent version update

[Ericson2314]

So work took me away from this, but the main problem might be that the script that generates the header during include was not in extra dist. There was a commit fixing this, but of course fetchpatch doesn't regenerate the tarball. Just fetching that file and seeing if any of the fetchpatch is actually needed is the next thing I'm going to try.