Nixpkgs: Installing NixOS in a container with a non-NixOS host is painful

[o11c]

Recently I installed NixOS into a systemd-nspawn container on a Debian host. I found this was quite painful.

I was using Debian stretch (testing), but backported systemd and systemd-container 226 from sid (unstable) due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=792882 - anyone wanting to install NixOS under systemd-nspawn should make sure systemd-nspawn is working first with some simpler system (e.g. by debootstraping jessie (stable))

Links to not-very-useful documentation:

• https://nixos.org/wiki/Install_NixOS_on_Linode - way too many steps
• https://nixos.org/wiki/How_to_install_NixOS_from_linux - gave the download and unsquash instructions, but then gives unnecessary commands to edit init to start bash instead of systemd and chroot manually.
• https://nixos.org/wiki/Install_NixOS_as_OpenVZ_guest - links to obsolete tarballs and refers to commands that no longer exist.
• https://nixos.org/wiki/NixOS:Containers - has a useful configure.nix to disable grub, but all the commands require a working nix host

Steps to reproduce:

• Create /etc/static/os-release on the host because Nix ships absolute symlinks, which is a bug. Per os-release(5), if /etc/os-release is a symlink, it must be relative. While NixOS doesn't use /usr/lib/os-release, the same principle applies (and there are multiple symlinks to follow - all need to be relative). This bug is obscured when the host is also NixOS because it also contains a (wrong!) /etc/static/os-release.
• Download http://nixos.org/releases/nixos/latest-iso-minimal-x86_64-linux
• Extract the squashfs from the ISO.
• Then extract the contents from the squashfs. It would be awesome if a runnable NixOS system could be available from a tarball (which does not require tools that probably weren't installed) instead of requiring these steps; the only tarballs I found were 4 years old on the OpenVZ page.
• Use systemd-nspawn to start the contents of the squashfs, and bind-mount the path you want to install the real NixOS system to /mnt. Entering it in any other way will clobber the hosts's UTS and mount namespaces - nixos-install doesn't form its own until far too late.
• When systemd bails out, press enter to use the rescue shell, that is enough.
• Copy /etc/resolv.conf from outside the container to inside it.
• Create /mnt/etc/nixos/configuration.nix with boot.isContainer = true;
• Run nixos-install. Note that if /mnt/ is not a mount point, nixos-install will bail out for no good reason, there should probably be a --force if people want to mv it later.
• Leave the squashfs container, you're done with it.
• Create sbin/init in the real container as a relative symlink to ../nix/var/nix/profiles/system/init. If you were fancy, you could do some sort of menu that lets you choose a profile.
• Create /etc/profile.local containing if [ "$TERM" = vt102 ]; then TERM=xterm; fi to fix systemd's braindead agetty settings.
• Start your new NixOS container with systemctl -bD $container_path (or machinectl start $machine_name or systemctl start systemd-nspawn@$machine_name if it's in /var/lib/machines) and enjoy!

[o11c]

Looks useful for the first step: https://hydra.nixos.org/job/nixos/release-14.12/nixos.system_tarball_pc.x86_64-linux

[rhendric]

As of this writing, this has gotten pretty simple (although it's still a few steps away from plug-and-play). Many thanks to those who came before me; here's what I learned from them and a little trial-and-error:

Pick your build from this list. Choose the last successful job, copy the URL of its build product, and use it below.

url=https://hydra.nixos.org/build/62686007/download/1/nixos-system-x86_64-linux.tar.xz
name=my-nixos
path=/var/lib/machines/$name
sudo machinectl pull-tar --verify=no $url $name
sudo mkdir $path/etc
sudo touch $path/etc/os-release
sudo mkdir $path/sbin
sudo ln -s /init $path/sbin/init

As with every nspawn container, you can configure it by creating an /etc/systemd/nspawn/$name.nspawn (see man page). Here's mine—this is the simplest way to get internet access from inside the container:

[Exec]
PrivateUsers=no

[Files]
Bind=/home

[Network]
Private=no
VirtualEthernet=no

Now you're ready to run:

sudo machinectl start $name
sudo machinectl shell $name
sudo machinectl stop $name

Before you run your first nixos-rebuild, if your host is running an sshd, you may want to add services.openssh.enable = false; to your configuration.nix to prevent a port conflict.

Tested on Arch Linux, but I don't see why it wouldn't work on other systemd distros.

[florianjacob]

Oh, so not being able to downloading stuff from hydra was fixed, yay! That indeed makes things much easier already.

I tried it with building my own containerTarball, but I only fixed /etc/os-release and couldn't figure out the /init part. Have to test this soon!

[florianjacob]

I created a PR for the os-release issue at #35364 , @rhendric could you please take a look? I hoped this could take the /etc/os-release part away from the nspawn setup, but I'm not sure anymore.

[florianjacob]

The nspawn code searching for the init system, which does not check /init but /sbin/init, can be found at https://github.com/systemd/systemd/blob/476f65f98a60b070bfec85842f220f0d37819f7f/src/nspawn/nspawn.c#L2477

[florianjacob]

We might be able to work around needing to modify the init part of the tarball by adding this to the Exec section:

[Exec]
Boot=no
Parameters=/init

The /init link itself is created at: https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/profiles/docker-container.nix#L25

Using the storeContents attribute, we could either:

a) change the link from /init to /sbin/init
b) add the /sbin/init link on top for both docker and lxc, or only the lxc image.
c) if the [Exec] workaround works, leave it as is, and document the remaining required setup e.g. in the https://nixos.wiki

[bchallenor]

Is there any way Hydra could automatically push this image to the Docker Hub?

[florianjacob]

The [Exec] workaround works! So now if #35364 will be merged, we're down to no modifications required anymore!

[rhendric]

Hm, not quite. It seems that using Boot=no does remove the need for /sbin/init (it also removes the need for /etc/os-release, so #35364 isn't even necessary), but it replaces it with a need for a /usr directory (see nspawn.c), which I had to manually create.

(Sorry for taking so long to get around to this; I tried a few times earlier but my free time always seemed to intersect with Hydra being down.)

[florianjacob]

That is really strange, it worked for me as described in #35364. But I see from the code it shouldn't have. @rhendric can you elaborate what happened on your system?

We probably should create our own base as an replacement for profile/docker.nix using /sbin/init instead of /init, or change it in the docker thing, so that we can boot=yes and land in this check: https://github.com/systemd/systemd/blob/2e10cc5649193eef5117fa5186869fbb74b0481e/src/basic/stat-util.c#L148

[florianjacob]

I thought maybe it's because I tried on NixOS which currently ships the somewhat outdated systemd v214, but no, that code is also in there. Why did it work for me, then? :thinking:

[rhendric]

Transcript:

$ name=nixos-test-charlie
$ path=/var/lib/machines/$name
$ sudo machinectl pull-tar -q --verify=no https://hydra.nixos.org/build/71761700/download/1/nixos-system-x86_64-linux.tar.xz $name
Content too large.
$ printf %s '[Exec]
> Boot=no
> Parameters=/init
> ' | sudo tee /etc/systemd/nspawn/$name.nspawn
[Exec]
Boot=no
Parameters=/init
$ sudo machinectl start $name
Job for [email protected] failed because the control process exited with error code.
See "systemctl status [email protected]" and "journalctl -xe" for details.
$ systemctl status [email protected] | cat
● [email protected] - Container nixos-test-charlie
   Loaded: loaded (/usr/lib/systemd/system/[email protected]; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-03-23 12:48:53 EDT; 1min 20s ago
     Docs: man:systemd-nspawn(1)
  Process: 22908 ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-veth -U --settings=override --machine=nixos-test-charlie (code=exited, status=1/FAILURE)
 Main PID: 22908 (code=exited, status=1/FAILURE)
   Status: "Terminating..."

Mar 23 12:48:53 adam systemd[1]: Starting Container nixos-test-charlie...
Mar 23 12:48:53 adam systemd-nspawn[22908]: Directory /var/lib/machines/nixos-test-charlie doesn't look like it has an OS tree. Refusing.
Mar 23 12:48:53 adam systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Mar 23 12:48:53 adam systemd[1]: [email protected]: Failed with result 'exit-code'.
Mar 23 12:48:53 adam systemd[1]: Failed to start Container nixos-test-charlie.
$ 

My host OS is Arch Linux on x86_64, in case it matters.

[danbst]

For reference, https://github.com/erikarvstedt/extra-container project removed this pain for me. I was able to launch NixOS container on Ubuntu 17.10. I've documented 2 main problems in https://github.com/erikarvstedt/extra-container/issues/1, of which one is NixOS problem (the check for /etc/static/os-release), and one is extra-container problem (registers service units in wrong directory, which exists only in NixOS).

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.