Nixpkgs: Cross-Compilation: State of the union for x86_64-linux binaries on darwin

[periklis]

Issue description

During the last two community of practice meetups of the Munich NixOS group, @lo1tuma, me & @globin analyzed the state of the union of cross-compilation for x86_64-linux binaries on a darwin host machine. The main driver behind the analysis is to use cross-compilation with dockerTools.buildImage.

Management summary:

• It is currently not yet possible to cross-compile binaries for the target platform x86_64-linux on a x86_64-darwin host platform.
• The main issue is that cross compilation depends on a host libc and in particular on darwin's Libsystem headers and dylibs modulo rpath-modifications with any nix-store dependencies.
• We explored three approaches:
  
  
  
  • Used darwin.xcode as codified in nixpkgs. Currently not possible due to nix issue #619.
  
  
  • Used darwin.usr-include is missing dylibs.
  
  
  • Extracted the Libsystem boostrap stage from stdenv.darwin into an additional package darwin.libsystem-inpure. Results in a mass rebuild if we add this to the buildInputs of cc-wrapper. More testing needs to finalize this approach.
  
  

Solution proposals in order of favor:

• Extract Libsystem from stdenv.darwin as an additional package for gccCrossStageStatic.
• Merge fix for #619 and use the older MacOSX SDK 10.9's Libsystem.

Further improvements for cross-compilation:

• More precise documentation of the crossSystem attribute set, e.g. libc is a disturbing field name because it is used for the libc of the host and not of the target system.
• Provide documentation or example code on how to automate all the isXXXX-fields in the crossSystem attribute set.

Steps to reproduce

1. Define crossSystem for your nix-build's default.nix:

let
  targetSystem = rec {
    config = "x86_64-linux-gnu";
    arch = "x86_64";
    withTLS = true;
    libc = "libSystem"; # This is the libc of your hostsystem
    platform = (with buildPkgs.lib.systems.platforms; pc64 // { kernelMajor = "2.6"; });
    openssl.system = "linux-generic64";
    isDarwin = false;
    isWindows = false;
    isCygwin = false;
    isLinux = true;
  };

  buildPkgs = import ../nixpkgs {};

  targetPkgs = import ../nixpkgs {
    crossSystem = targetSystem;
  };
in
targetPkgs.hello

1. Building your package, follows the dependency graph inside stdenv.cross:

[BuildPackages]                            [TargetPackages] 
stdenv.cross ----> gccCrossStageFinal ---> hello
                   |
                   |
                   |
                   ---> libcCross ---> gccCrossStageStatic --> crossSystem.libc e.g. Libsystem

Example code for Libsystem

{stdenv
, localSystem
, bootstrapFiles ? let
  fetch = { file, sha256, executable ? true }: import <nix/fetchurl.nix> {
    url = "http://tarballs.nixos.org/stdenv-darwin/x86_64/c4effbe806be9a0a3727fdbbc9a5e28149347532/${file}";
    inherit (localSystem) system;
    inherit sha256 executable;
  }; in {
    sh      = fetch { file = "sh";    sha256 = "1b9r3dksj907bpxp589yhc4217cas73vni8sng4r57f04ydjcinr"; };
    bzip2   = fetch { file = "bzip2"; sha256 = "1wm28jgap4cbr8hf4ambg6h9flr2b4mcbh7fw20i0l51v6n8igky"; };
    mkdir   = fetch { file = "mkdir"; sha256 = "0jc32mzx2whhx2xh70grvvgz4jj26118p9yxmhjqcysagc0k7y66"; };
    cpio    = fetch { file = "cpio";  sha256 = "0x5dcczkzn0g8yb4pah449jmgy3nmpzrqy4s480grcx05b6v6hkp"; };
    tarball = fetch { file = "bootstrap-tools.cpio.bz2"; sha256 = "0ifdc8bwxdhmpbhx2vd3lwjg71gqm6pi5mfm0fkcsbqavl8hd8hz"; executable = false; };
  }
}:

let
  inherit (localSystem) system;

    libSystemProfile = ''
    (import "${../../../stdenv/darwin/standard-sandbox.sb}")
  '';

  binShClosure = ''
    (allow file-read* (literal "/usr/lib/libncurses.5.4.dylib"))
  '';

  bootstrapTools = derivation rec {
    inherit system;

    name    = "bootstrap-tools";
    builder = bootstrapFiles.sh; # Not a filename! Attribute 'sh' on bootstrapFiles
    args    = [ ../../../stdenv/darwin/unpack-bootstrap-tools.sh ];

    inherit (bootstrapFiles) mkdir bzip2 cpio tarball;

    __sandboxProfile = binShClosure + libSystemProfile;
  };

in stdenv.mkDerivation {
  name = "libsystem-impure";

  buildCommand = ''
    mkdir -p $out
    ln -s ${bootstrapTools}/lib $out/lib
    ln -s ${bootstrapTools}/include-Libsystem $out/include
  '';
}

and use in all-packages.nix:

libsystem-impure = callPackage ../os-specific/darwin/libsystem-impure {
      localSystem = buildPlatform;
} ;

replace all darwin.xcode occurrences with darwin.libsystem-impure in all-packages.nix.

Technical details

• System: macOS 10.12.6
• Nix version: nix-env (Nix) 1.11.13
• Nixpkgs version: 17.09pre-git
• Sandboxing enabled: -

cc @Ericson2314 & @fpletz

[Ericson2314]

I think the attributes are correct just being used incorrectly. This should be much easier to fix. On commute + phone now, but can discuss soon.

[copumpkin]

I think the key component here is _running_ Linux binaries on a Darwin machine, right? @puffnfresh is doing some work on transparent VMs for that, but cross-compilation doesn't seem like the core issue here to me.

[copumpkin]

I also don't understand the darwin.xcode stuff, which isn't really used very much at all.

[puffnfresh]

Yes, I am trying to solve the dockerTools problem for my team. So far I have got HyperKit working within Nix and have a script to configure it as a remote builder. You can find my work here:

https://github.com/puffnfresh/nixpkgs/tree/feature/hyperkit-builder

I would like to try a few different things:

1. Write a better wrapper around starting the remote builder container, at the moment you have to know/care about disk size and DHCP
2. VPNKit might be useful
3. I'd like to get /nix/store via a 9P mount (I have a version of Diod which can do 9P on Darwin) then combine that with OverlayFS

Then I'd like to think about turning the remote builder into a:

1. Build substituter
2. Backend for runInLinuxVM

I'm confident that HyperKit is the appropriate tool for producing x86_64-linux on x86_64-darwin.

[globin]

This is about cross-compiling linux binaries on darwin, not about running them.

[periklis]

@copumpkin
darwin.xcode is afaik a bridge to provide a libsystem for packages like gccCrossStageStatic, which are built by apple. I suppose because nobody except apple can built the libsystem from source.

@puffnfresh
Thx for the link to your HyperKit work. I'll take a look on it. However, can you elaborate why cross-building linux binaries on darwin with the nixpkgs.stdenv.cross means is (not) enough for producing linux binaries for docker images?

[puffnfresh]

@periklis dockerTools has stuff around building root layers, which use runInLinuxVM. It should hopefully be possible if you don't use the root layer building.

[periklis]

@puffnfresh i had a look on your hyperkit branches and it is indeed a very interesting approach to run a vm as nix-builder machine through xhyve. This approach makes cross-compilation totally obsolete, but constrains the handling of total resources to build, doesn't it?

Did you collect any cross-compilation experience in the first place and then switched to the hyper-kit approach? ooo(i saw a stale cross-compilation on your repo)

[copumpkin]

@globin but my point is that if you're building Docker images with the current codebase for doing that, you're also _running_ Linux binaries to produce the images. And once you're running Linux binaries, you can build Linux binaries more easily

[lo1tuma]

@copumpkin That is only the case when you build an image with using runAsRoot or am I missing something?

[copumpkin]

Oh, you might be right. I haven't looked at the docker builder in a long time. I'd love it if we could build those images with no VM involvement!

[lo1tuma]

The following example builds successfully on macOS:

{ pkgs ? import <nixpkgs> {} }:

let
  inherit (pkgs) dockerTools;
in
  dockerTools.buildImage {
    name = "hello";
    tag = "latest";
    config = {
      Cmd = [ "${pkgs.hello}/bin/hello" ];
    };
  }

I can even load the image via docker load < ./result, but I can’t run it because the hello binary in the image is a macOS binary.

[copumpkin]

@lo1tuma oh, I see. So in that case just doing the following:

{ pkgs ? import <nixpkgs> {} }:

let
  inherit (pkgs) dockerTools;
  linuxPkgs = import <nixpkgs> { system = "x86_64-linux"; };
in
  dockerTools.buildImage {
    name = "hello";
    tag = "latest";
    config = {
      Cmd = [ "${linuxPkgs.hello}/bin/hello" ];
    };
  }

Should be fine if binaries are available in the cache. And with @puffnfresh's builder, it should build them seamlessly in the background Linux VM similarly to how Docker for Mac works. Cross compilation would also be nice but feels like a much gnarlier problem.

Unfortunately, the above snippet does not work on my Nix on macOS because it claims that Docker is unsupported on Darwin. Did you do something special to make our docker package work on macOS there?

[LnL7]

The dockerTools don't depend on docker AFAIK

[copumpkin]

@LnL7 I just tried evaluating the snippet in my post above and it complained that docker was unsupported. Not sure what depends on it:

error: Package ‘docker-17.06.0-ce’ in /nix/store/zdzvl6fk60cw4izxfakhxiz17aipxfd1-nixpkgs-17.09pre111456.d4ef5ac0e9/nixpkgs/pkgs/applications/virtualization/docker/default.nix:165 is not supported on ‘x86_64-darwin’, refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowBroken = true; }
to ~/.config/nixpkgs/config.nix.

(use ‘--show-trace’ to show detailed location information)

[LnL7]

Must have changed recently then.

[puffnfresh]

dockerTools switched to using Docker to pull images recently. There were problems around image URLs.

[periklis]

Alternative approach by @puffnfresh: #29628

[Ericson2314]

@LnL7 and I started working on linux->Darwin cross compilation.

[copumpkin]

You mean building Darwin on Linux or building Linux on Darwin?

[periklis]

@Ericson2314 Great news! My intention was only to cross-reference to @puffnfresh approach.

[periklis]

@Ericson2314 do you have a jobset for the linux->darwin cc to watch?

[Ericson2314]

Oh sorry meant Darwin->Linux. No jobset just yet, but https://github.com/Ericson2314/nixpkgs/tree/darwin-to-linux is a branch

[periklis]

@Ericson2314 Your WIP starts with https://github.com/Ericson2314/nixpkgs/commit/d8243d3befbe5b33749b39eb48442dfa227242a2 on top of #29396 and #26805 right?

[Ericson2314]

Indeed! All those branches will end up being rebased, but that's the order.

[Ericson2314]

And yes with a special substituter, the @puffnfresh approach and mine could dovetail nicely

[matthewbauer]

@Ericson2314 Is the "special" substituter inevitable for macOS cross? I'd like to be able to do "true" cross compile stuff on macOS but everything seems to lead to @puffnfresh's Linux VM route. That can probably work but seems a little bit like cheating. Ideally macOS cross compiling should work just like Linux cross compiling does.

[Ericson2314]

@matthewbauer The special substituter stuff is only necessary to use a native cache for cross builds.

• Cross compilation from macOS with GCC should already work (https://github.com/input-output-hk/nix-hs-hello-windows/tree/build-more).
• Cross compilation with LLVM is tracked in https://github.com/NixOS/nixpkgs/issues/36867.
• macOS->iOS cross compilation with prebuilt libraries via the SDK's tbd files I am working on now.
  
  
  
  • adaa110a72175c4f841722ab7d2bee7564656e91 is quite useful to avoid infinite recursion, but is unfortunately currently stuck on staging.
  
  

[periklis]

@Ericson2314 Actually there was an experimental branch of yours obsidiansystems/darwin-to-linux from last Nixcon, which was fruitful thought on the missing parts for darwin-to-linux cross-compilation, but i can't find it anymore.

[LnL7]

@periklis These are my changes based on John's branch darwin-to-linux. I got stuck with the kernel build because of case sensitivity issues, etc.