Nixpkgs: Installer builds derivations on host store

[bennofs]

Issue description

The current installer builds the derivations on the store of the installer's image, which means that if you install a large config the install will fail since there is not enough space available.

Tested with nixpkgs unstable minimal install image.

[copumpkin]

This is my fault, sorry. I welcome suggestions for how to fix without duplicating all the logic that led to this change in the first place! 😦

Edit: this is the PR in question: https://github.com/NixOS/nixpkgs/pull/23026

[bennofs]

Perhaps we can create a /new-root/nix-storage and mount that on top of /nix/store using unionfs so that we use the new disk as temporary storage?

There's two disadvantages of that approach though that I can see:

1. The target disk needs to have at least twice the amount of storage as the final system profile requires
2. There will be an unnecessary copy from /nix/store to /new-root/nix/store, even though /nix/store is already mostly on the target disk.

[copumpkin]

There's also the new Nix 1.12 custom store location stuff, which will probably help us.

[emmanuelrosa]

Yes! I've been pondering this issue myself. My work-around has been to install as little as possible, and then once installed add the remaining packages. That's for a release-17.03 install; when I tried an unstable ISO I couldn't get it to install at all with only 2GB of RAM available.

I've thought about...

1. Temporarily utilizing the swap partition by placing a filesystem on it and making it the rw part of the unionfs.
2. For USB flash-drive installs, utilize a partition on the remaining space as the rw part of the unionfs.

But to generalize the concept, perhaps it's possible to pass a parameter via the boot-loader to tell whatever script sets up the Nix store unionfs to mount the filesystem at the specified block device.

[Shados]

I tend to do frequent installations to little memory-constrained VPS nodes in different locations (for e.g. monitoring agents, DNS nodes), and this issue bites me in the ass every single time >.>. I've been working around it by temporarily reducing my configuration from the intended final config (which I have prepared beforehand, as they're all pretty similar) to a minimal one, then building it to the final one after reboot.

[copumpkin]

My apologies. I'll try to improve this during the NixCon hackathon early next week! It won't be my primary goal for the hackathon but it's something that's been annoying me for a while so it'll probably get done 😄

[copumpkin]

As you've probably noticed (from the latest nix-devel email thread on it) I didn't work on this at NixCon, but it's still pretty high in my priority queue of things to do. If someone else wants to take a stab at it though, I won't be offended 😄

[danbst]

BTW, the idea to use nixUnstable in nixos-install should deserve some thought:

• the Nix db should be backwards compatible, so no issues when running nix 1.11 after reboot
• no need to write extra code, when it can be done in several 1.12 commands
• battle test on fresh users. This is debatable though, but nixos-unstable users are warned about possible hiccups

[copumpkin]

We already do use nixUnstable inside nixos-prepare-root, which nixos-install uses, so I think we're already set for this. Could definitely just set up a remote store for this.

[vcunat]

Claimed to be fixed by #34636.

[tycho01]

did #34636 make it into 18.03? I'm running into this on that.
has anyone who ran into this managed to verify this fix?

edit: fwiw I got a bit further by allocating 2 GB to the boot partition rather than the 512 MB the NixOS installation instructions recommended.