Nixpkgs: busybox in bootstrap-tools doesn't work in WSL

[hsenag]

Issue description

Windows Subsystem for Linux doesn't currently support 32-bit binaries:
https://github.com/Microsoft/BashOnWindows/issues/390

Since x86_64 bootstrap-tools includes the i686 busybox, any build that requires it fails:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/linux/bootstrap-files/x86_64.nix#L2

Could it be changed to use an x86_64 specific busybox? With that hacked locally I got a lot further using WSL with nix. I'm happy to produce a patch myself but I'm not sure what the official way to update bootstrap-files is.

Steps to reproduce

Do something that requires bootstrap-tools.

Technical details

• System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
  Ubuntu running under WSL (Windows 10 Creators Update):

ganesh@Priory:/mnt/c/msys64/home/ganesh/nixpkgs/general$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty

• Nix version: (run nix-env --version)
  nix-env (Nix) 1.11.8
• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
  "17.09pre105126.efe12d6658"

[andyli]

Same issue here. I'm also running WSL with Windows 10 Creators Update, but I've a clean install of Xenial instead of Trusty:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.2 LTS
Release:        16.04
Codename:       xenial
$ nix-env --version
nix-env (Nix) 1.11.9
$ nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
"17.09pre106045.7369fd0b51"

[pikajude]

@hsenag can you post what you have working at the moment? I've also run into this issue

[hsenag]

I dropped the import i686.nix and replaced it with this:

   busybox = import <nix/fetchurl.nix> {
      # made by just building busybox in an existing x86_64 nix install
     url = http://urchin.earth.li/~ganesh/temp/busybox;
     sha256 = "1wdhy09h4csm7vjk05s5sc8a9wypbv94ln4ikzjvyayrminky5sg";
     executable = true;
  };

You're welcome to refer to my URL if you want (no promises about how long it'll stay) or you can do something similar.

[puffnfresh]

I want to reference the x86_64 version of busybox in here:

https://github.com/NixOS/nixpkgs/blob/042ca21c812ae3c3fd52f3fdec41699b193dcfbf/pkgs/stdenv/linux/bootstrap-files/x86_64.nix#L6

But http://tarballs.nixos.org/stdenv-linux/x86_64/4907fc9e8d0d82b28b3c56e3a478a2882f1d700f/busybox is giving 404 - is that where it should be?

[dtzWill]

Partially to avoid this issue, musl bootstrap uses a 64bit busybox. Here's the latest info:

https://github.com/NixOS/nixpkgs/pull/36384/files#diff-e5cfbf7cb5224d730a5ce732c0850a0dR3

If you try it, please let us/me know if it works for you!

[eqyiel]

@dtzWill thank you, I had the same problem as @hsenag and using busybox from x86_64-musl.nix worked for me.

Changes in 0e49a63^..0e49a63
1 file changed, 3 insertions(+), 2 deletions(-)
pkgs/stdenv/linux/bootstrap-files/x86_64.nix | 5 +++--

modified   pkgs/stdenv/linux/bootstrap-files/x86_64.nix
@@ -1,5 +1,6 @@
-# Use busybox for i686-linux since it works on x86_64-linux as well.
-(import ./i686.nix) //
+# i686 doesn't work on WSL right now.  Use the x86_64 busybox from x86_64-musl instead.
+# See https://github.com/NixOS/nixpkgs/issues/24954#issuecomment-370871154
+(import ./x86_64-musl.nix) //

 {
   bootstrapTools = import <nix/fetchurl.nix> {

I wonder if the default should be changed.

[eqyiel]

Here's another workaround: https://stackoverflow.com/a/49405605

QEMU and binfmt support light the way :)

https://github.com/microsoft/wsl/issues/2468#issuecomment-374904520

After reading that the WSLInterop between WSL and Windows processes used binfmt, I was tinkering with QEMU to try some ARM development, and incidentally discovered how to get 32-bit support working.

Edit: requires "Fall Creators Update", 1709, build 16299 or newer

Install qemu and binfmt config:

sudo apt install qemu-user-static
sudo update-binfmts --install i386 /usr/bin/qemu-i386-static --magic '\x7fELF\x01\x01\x01\x03\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x03\x00\x01\x00\x00\x00' --mask '\xff\xff\xff\xff\xff\xff\xff\xfc\xff\xff\xff\xff\xff\xff\xff\xff\xf8\xff\xff\xff\xff\xff\xff\xff'

You'll need to reactivate binfmt support every time you start WSL:

sudo service binfmt-support start

At this point, Nix should be able to use the i686 bootstrap tools.

[dtzWill]

Neat! Glad the musl tools helped, but the other approach might be preferred in some situations and is interesting regardless! :+1:

[matthewbauer]

Is there an easy way to detect WSL? Lots of places we assume that x86_64-linux can also run i686-linux. On WSL it sounds like that won't work (but it can run x86_64-windows!) Maybe we can use the Musl-based bootstrap in that case.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[drewrisinger]

Still important to me. Workaround from @eqyiel with qemu for running the 32-bit busybox worked.

For detecting if in WSL, a quick google found this thread: https://github.com/microsoft/WSL/issues/423

Output of some of the commands from that:

$ cat /proc/version
Linux version 4.4.0-19041-Microsoft ([email protected]) (gcc version 5.4.0 (GCC) ) #1-Microsoft Fri Dec 06 14:06:00 PST 2019
$ cat /proc/sys/kernel/osrelease 
4.4.0-19041-Microsoft
$ env | grep -i wsl
WSL_DISTRO_NAME=Ubuntu
WSLENV=VSCODE_WSL_EXT_LOCATION/up

[matthewbauer]

This should just be an issue with WSL1, as WSL2 has i686 support.

[drewrisinger]

@matthewbauer that sounds right to me. I recently upgraded to Win 20.04 & tried on a fresh Ubuntu WSL install, assuming it would upgrade to WSL2 automatically, didn't realize I missed a few steps. trying to fix now.

EDIT: fixed by upgrading to WSL2. For future reference, the way to check is (via command-line): wsl -l -v.

[TethysSvensson]

The way I see it, we have a few options for closing this issue:

• We could try to pursue making the glibc.x86_64-linux bootstrap actually be a x86_64 binary.
  
  
  
  • Would there be other benefits to doing this or would it only matter for WSL1?
  
  
  • Would this even fix the issue, given @matthewbauer's comment?
  
  
• We could try to automatically detect WSL1 and pick a musl bootstrap instead of a glibc bootstrap.
  
  
  
  • This suffers from the same issues wrt @matthewbauer's comment.
  
  
• We could accept that WSL1 is not very well supported out-of-the-box.
  
  
  
  • Given that Microsoft recommends WSL2 for most use-cases, this seems like an okay solution.
  
  
  • If we pick this option, we might consider improving documentation with the proposed work-arounds somewhere, i.e. binfmts+qemu, upgrading to WSL2 or potentially trying with a musl bootstrap.
  
  

[drewrisinger]

Personally, I'm OK with 3. Hopefully this won't be an issue in the future as WSL2 should be the default, and Nix works out-of-the-box with WSL2.

We could make a page on the nixos.wiki detailing the above WSL1 workaround.

[TethysSvensson]

Would the WSL section of the page Nix Installation Guide be the right place perhaps?

[drewrisinger]

I'd never noticed that page. It looks like that heading is referring just to the "Nix store", but another heading on the same page would probably be appropriate. The only caveat I see is that page mostly refers to installation of the _Nix_ tools itself, and not to the _nixpkgs_ programs that are actually run (e.g. busybox).

[TethysSvensson]

Would somebody more familiar with WSL be up for writing something on the wiki summarizing the work-arounds found in this issue?

[drewrisinger]

I wrote up a basic guide at https://nixos.wiki/wiki/Nix_Installation_Guide#Windows_Subsystem_for_Linux_.28WSL.29. Feel free to edit if you'd like. I think this issue might be able to be closed now.