Nixpkgs: Why can't Linux be built remotely?

[pikajude]

I've defined this package in my config.nix:

  {
    nix-build-remote = pkgs.writeScriptBin "nix-build-remote" ''
      #!${pkgs.stdenv.shell}
      mkdir -p /tmp/build-remote-load
      chmod a+rwX /tmp/build-remote-load

      export NIX_BUILD_HOOK="${nix.out}/libexec/nix/build-remote.pl"
      export NIX_CURRENT_LOAD="/tmp/build-remote-load"
      exec "${nix.out}/bin/nix-build" "$@"
    '';
  }

$ nix-build-remote '<nixpkgs/pkgs/top-level/release.nix>' -A linux_4_4 -vvv
[lots of output]
starting build hook
hook reply is ‘decline’
building path(s) ‘/nix/store/lm0mf0v9fjgi18nmnjf05nfcjr06jsy7-linux-4.4.12-dev’, ‘/nix/store/ng0iig19fw85xqg0afqa3zjhg6pfw4gi-linux-4.4.12’
[more output]
building of ‘/nix/store/c9shhhq395xxrz791hq871h5m43bkyl3-linux-4.4.12.drv’: goal destroyed
error: a ‘x86_64-linux’ is required to build ‘/nix/store/c9shhhq395xxrz791hq871h5m43bkyl3-linux-4.4.12.drv’, but I am a ‘x86_64-darwin’

What causes this? It's very inconvenient to have to copy the derivation to the build box, build it there, and copy it back.

[edolstra]

This may be due to 1f84e43239bc2a707c30c0d7bca47801e4df5a78. You need to have a remote machine with the feature big-parallel.

[domenkozar]

We should emit a better error message.

[vcunat]

Typically you don't get any error message and it just silently builds locally. I did take me some time to figure out why (back in May).

[Gabriel439]

I just ran into this issue while using nixops to deploy a custom kernel to a virtual machine. Here is the minimal reproducing nixops config that caused the problem:

{ local = { config, pkgs, ... }: {
    config.deployment.targetEnv = "virtualbox";
    config.deployment.virtualbox.memorySize = 4096;
    config.deployment.virtualbox.headless   = true;
    config.boot.kernelPackages = pkgs.linuxPackages_4_4;
    config.nixpkgs.config.packageOverrides = pkgs:
    {
        stdenv = pkgs.stdenv // {
            platform = pkgs.stdenv.platform // {
                kernelExtraConfig = "CMA y" ;
           };
        };
    };
  };
}

Here's the rough chain of events:

• The host machine that you deploy from is OS X and the VM is Linux so nixops is smart and tries to build the Linux config and the Linux kernel on the VM using nix's support for remote builds
• The build for the Linux config on the VM succeeds
• However, the VM does not support the big-parallel system feature required by the Linux kernel derivation so the VM refuses to build the Linux kernel
• nixops then falls back to trying to build the Linux kernel locally on the host (OS X), failing with the exact same error described at the beginning of this issue

error: a ‘x86_64-linux’ is required to build ‘/nix/store/...-linux-4.4.12.drv’, but I am a ‘x86_64-darwin’

I was able to work around this by:

• Forking pkgs/os-specific/linux/kernel/manual-config.nix to ./manual-config-fork.nix and deleting the requiredSystemFeatures = [ "big-parallel" ]; line in my fork

diff --git a/pkgs/os-specific/linux/kernel/manual-config.nix b/tmp/test/manual-config.nix
index 8da8f20..1cc8f7f 100644
--- a/pkgs/os-specific/linux/kernel/manual-config.nix
+++ b/tmp/test/manual-config.nix
@@ -194,8 +194,6 @@ let
           $installFlags "''${installFlagsArray[@]}"
       '');

-      requiredSystemFeatures = [ "big-parallel" ];
-
       meta = {
         description =
           "The Linux kernel" +

• Adding the following buildLinux override to my original nixops config

diff --git a/vbox.nix b/vbox2.nix
index 1e209c5..1d34157 100644
--- a/vbox.nix
+++ b/vbox2.nix
@@ -5,6 +5,8 @@
     config.boot.kernelPackages = pkgs.linuxPackages_4_4;
     config.nixpkgs.config.packageOverrides = pkgs:
     {
+        buildLinux = pkgs.callPackage ./manual-config-fork.nix {};
+
         stdenv = pkgs.stdenv // {
             platform = pkgs.stdenv.platform // {
                 kernelExtraConfig = "CMA y" ;

With those two changes the Linux kernel now successfully builds on the VM instead of OS X.

Would it be possible for nix output improved diagnostics about why it's not selecting remote build machines? If people are okay with that proposal I might have time to contribute a pull request

[rasendubi]

I wonder why big-parallel exists as a _required_ system feature in the first place. I understand it's for selecting a more powerful host for the build, but I don't think that's really _required_, as Linux kernel can be built on the less powerful machine with a single thread, though taking an enormous amount of time. (Is there a definition of "big" after all?)

[vcunat]

I think it's simply because Nix ATM only provides support for _required_ features and no better way.

[rasendubi]

(triage) I think the question is answered? Should we extract action items and close the issue?

[edolstra]

The action item would be to add a preferredSystemFeatures attribute for Hydra and/or build-remote.pl.

[Gabriel439]

@edolstra Also: better diagnostics for the build-remote.pl script so that it explains why it skipped over machines listed in the remote systems configuration file

[domenkozar]

Note that it's being rewritten to C++ in https://github.com/NixOS/nix/pull/981

[grahamc]

I would propose the error message indicate the features it requires:

"a ‘%1%’ with the features ‘foo’, ‘bar’, ‘baz’ is required to build"

as a hint to the resolution.

[nh2]

Relevant: https://github.com/NixOS/nix/issues/2238

[tomberek]

(triage) close as answered?

[nh2]

See above:

Should we extract action items and close the issue?

The action item would be to add a preferredSystemFeatures attribute for Hydra and/or build-remote.pl.

That action item should be filed somewhere if we close this issue.