Nixpkgs: NixOS remote builder with cross compilation

[anna328p]

Issue description

Is there an established method of using a remote builder with a cross compiler?

I am attempting to run NixOS on a Raspberry Pi using the armv7 architecture, but there is very little software built for it in Hydra so much of the configuration has to be compiled. This compilation process has now been running for three days.

Is there a way to run a builder daemon on my x86_64 system to cross compile to ARM more quickly?

[volth]

nix-build -E "(import <nixpkgs> {}).pkgsCross.armv7l-hf-multiplatform.perl" (on x86_64) should compile for armv7

[matthewbauer]

Yeah you can have a NixOS config that is cross compiled. The trick is to set localSystem = { system = "x86_64-linux"; }; so that rebuilds still happen on your x86_64 machine. Something like this should work:

{ pkgs, lib, config, ... }:

# Build with:
#
# nix build '((import <nixpkgs/nixos/lib/eval-config.nix> { modules = [ ./configuration.nix ]; }).config.system.build.sdImage)'

{
  imports = [
    <nixpkgs/nixos/modules/installer/cd-dvd/sd-image-raspberrypi.nix>
  ];

  boot.loader.raspberryPi.enable = true;
  boot.loader.grub.enable = false;
  boot.kernelPackages = pkgs.linuxPackages_rpi;

  # Setup cross compilation.
  nixpkgs = {
    crossSystem = lib.systems.elaborate lib.systems.examples.raspberryPi;
    localSystem = { system = "x86_64-linux"; };
  };

  nix.distributedBuilds = true;
  nix.buildMachines = [ {
    hostName = "linux64.example.org";
    sshUser = "buildfarm";
    sshKey = "/root/.ssh/id_buildfarm";
    system = "x86_64-linux";
  } ];

  # Enable tools for remote installation.
  services.openssh.enable = true;
  networking.wireless.enable = true;
  security.sudo.enable = true;
  hardware.enableRedistributableFirmware = true;

  # Setup headless mode.
  systemd.enableEmergencyMode = false;
  documentation.enable = false;
  services.nixosManual.showManual = lib.mkForce false;

}

You need to copy it to the installation afterwards and setup the ssh keys for remote buildings. This means that every time you run nixos-rebuild you will build on your x86_64 machine. Unfortunately, you won't be able to build natively because Nix doesn't recognize that x86_64 -> armv7l compilation is the same thing as armv7l -> armv7l compilation.

[FRidh]

Maybe it's still too early for it but otherwise I think it would be good to add a section on cross-compiling NixOS to the NixOS manual.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/remote-cross-compiled-builds-for-macos-darwin-from-nixos-remote/5963/1

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[Atemu]

Still important to me.

[anna328p]

for reference, I ended up moving to the Raspberry Pi 4 platform and using aarch64 as both the build and host architecture. I also needed to add extra-platforms = aarch64-linux arm-linux to /etc/nix/nix.conf. this meant that nix fetched prebuilt binaries from Hydra and I did not have to cross-compile anything.