Nixpkgs: armv7l images are unable to run nixos-rebuild, nix, etc.

[endgame]

Issue description

Not sure this is the correct repo. Will refile if needed, just let me know.

The wiki at https://nixos.wiki/wiki/NixOS_on_ARM links to http://nixos-arm.dezgeg.me/installer maintained by @dezgeg , and the armv7l image is affected by https://github.com/NixOS/nix/issues/2393 , meaning that it gets wedged and you cannot upgrade.

(Aside: The aarch64 image has nix 2.1.1, which is unaffected, but the disk image does not match the checksum in SHA256SUMS.)

Steps to reproduce

Download https://www.cs.helsinki.fi/u/tmtynkky/nixos-arm/installer/sd-image-armv7l-linux.img , dd it to a micro sd and stick it in a pi 3b. Most invocations of nix will hit something that wants an xz archive, and from there it's impossible to proceed due to the linked bug above.

Workaround

It is possible (if you have a day or so) to get a bootstrap toolchain from outside nix, install it, build nix and all its dependencies by hand, and then try to use an external nix to fix the nixos install. I have attached my notes to this issue: upgrade-nix.txt

[JasonSwindle]

@dezgeg Any luck on this?

[qolii]

Oof, I think I am hitting this. It wedges up here (for me) every time:

building '/nix/store/j4z5c8afgxclbdidfs5a9gpfvk7byjws-bootstrap-tools.tar.xz.drv'...
fixed-output derivation produced path '/nix/store/qdrjfapishmpp5f5i8dq22aynwrcl7aj-bootstrap-tools.tar.xz' with sha256 hash '1ndnr0lp0qzdkf6p7zp64hg7l2msgjpyqcmyq25y256h3ipinqy2' instead of the expected hash '05ayki2kak3i5lw97qidd5h9jv00dmlhx9h7l771bj331yamyqdn'
cannot build derivation '/nix/store/wlmx3k9zdqc57r3w7wc3nd65zdc5x2ry-bootstrap-tools.drv': 1 dependencies couldn't be built

Could this be worked around using an older arm7 image? Might you have one laying around, @dezgeg?

[endgame]

If you have a lot of patience, can you try the bootstrapping instructions I put in the .txt link at the top of the issue? That builds a newer nix from source.

[qolii]

@endgame, looking over those instructions, it looks involved! I'm not sure I have the time to put into it.

Do you mean though that the issue is fixed in nixpkgs master? In that case, would a new arm image be all that is needed? If so, how are these images made? I have a working arm7 machine if that is helpful.

[qolii]

Ugh, sorry. I just saw the instructions. I will give them a try when my poor little arm board has finished its (unrelated) nixos-rebuild. It may be a minute.

[endgame]

I didn't know about the build-your-own-image instructions. I was talking about the long and slow manual build of nix. At least it looks like you have a way forward.

[furrycatherder]

@dezgeg I tried setting up an aarch64 build host with qemu:

1:  NixOS - Default
Enter choice: 1
1:  NixOS - Default
Retrieving file: /extlinux/../nixos/4n8f7w3zvin3fxvs5km8qmypc9a92lca-initrd-initrd
6853747 bytes read in 119 ms (54.9 MiB/s)
Retrieving file: /extlinux/../nixos/zbpbdvcazhh44sfzv9rgy9wllnab66v5-linux-4.18.7-Image
30059008 bytes read in 493 ms (58.1 MiB/s)
append: systemConfig=/nix/store/sy8zfls2p4g2p7wjls4vyw487a794bkz-nixos-system-nixos-19.03.git.56b9f6fc8e1 init=/nix/store/sy8zfls2p4g2p7wjls4vyw487a794bkz-nixos-system-nixos-19.03.git.56b9f6fc8e1/init loglevel=7 cma=32M console=ttyS0,115200n8 console=ttyAMA0,115200n8 console=tty0
Bad Linux ARM zImage magic!

Could you please regenerate the installer images?

[samueldr]

The aarch64 images are now generated on NixOS' hydra instance

• https://hydra.nixos.org/job/nixos/release-18.09-aarch64/nixos.sd_image.aarch64-linux
• https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux

[furrycatherder]

@samueldr I'm having the same problem with the Hydra builds. Is there an example qemu command I could try?

[samueldr]

I never have used the sd_image with qemu, but I have used the (relatively) new UEFI iso image for aarch64. Using this, and a valid aarch64 "bios" for qemu (look for the second output, its name should end with -fd) I was able to boot an aarch64 qemu, either with emulation (on x86_64) or even virtualisation (on an aarch64 host).

[qolii]

Hm, I wonder if someone can help me. My machine finally finished its update! And now I am trying to make my own sd-image.

My problem is with lkl, which refuses to build on armv7. After adding armv7l-linux to its list of platforms, I get the build error:

lib/hijack/hijack.c: In function 'fcntl':
lib/hijack/hijack.c:242:18: error: '__lkl__NR_fcntl' undeclared (first use in this function); did you mean '__lkl__NR_fcntl64'?
  return lkl_call(__lkl__NR_fcntl, 3, fd, lkl_fcntl_cmd_xlate(cmd), arg);
                  ^~~~~~~~~~~~~~~
                  __lkl__NR_fcntl64

Which seems maybe related to https://github.com/lkl/linux/pull/372? Their fix seems to specifically look for android though, which won't help of course.

Any ideas?

Thinking differently, the package needs lkl only to make use of the cptofs command inside the builder. Is there an alternative way to get this effect? I guess we cannot just mount the fs and cp things to it in the build environment?

[qolii]

(But I am also a little confused. How did this ever work to build the armv7 images??)

[qolii]

@copumpkin, in case you have any ideas! I must be missing something obvious.

[samueldr]

(But I am also a little confused. How did this ever work to build the armv7 images??)

It previously wasn't using lkl, but debugfs, hmmm, this would be an annoyance if there was a regression on an unsupported, but working, platform like armv7l; it's harder to work on, test, and fix.

[qolii]

Ooooh, thanks! Perhaps I can try building an image using a commit from just before that change was merged.

[qolii]

Success! @samueldr, I reverted the change you linked me to and rebuilt. The resulting image boots nicely on my hardware.

If anyone needs a working armv7 sd image, and knows of a place I could stash it, let me know!

[JasonSwindle]

@qolii I would love access to it, if possible. :)

[qolii]

Hi @JasonSwindle, sorry this took a long time. I've added to a release on my nixpkgs fork.

You should be able to just whack this on an SD card like the normal image.

[IreneKnapp]

Thanks very much, @qolii! I can confirm that your release works great for me. Strictly speaking, I tested it on an eMMC card rather than an SD card.

[Thra11]

I just encountered this issue, except on armv6l, not armv7l, so I wasn't able to avoid it by using @qolii 's image.
However, I thought I would point out that if you have another suitable build host with nix installed, you can use it to cross-compile a newer version of nix. For armv6l:

nix build nixpkgs.pkgsCross.raspberryPi.nix

or for armv7l:

nix build nixpkgs.pkgsCross.armv7l-hf-multiplatform.nix

You can then use nix-copy-closure or nix-store --export and nix-store --import to copy the required derivations onto the arm device. (At the moment, you get a few hundred MB of debug and host libs in your closure, but it's not really a problem in this instance)
This should be less involved and considerably faster (on my build host, the cross toolchain was actually fetched from cache.nixos.org, saving a lot of time) than the manual steps detailed by @endgame.

[misuzu]

cptofs is broken on armv7l when resulting filesystem is bigger than 2G.
Why not use -d option of mkfs.ext4 when creating image? lkl will not be needed anymore so less stuff to compile and maintain.