Nixpkgs: nixos-assimilate - Turn currently running system into NixOS

[aszlig]

This command would not only be very useful for users to switch to NixOS but also would get NixOps to support a large variety of different hosters. Also this would simplify the Hetzner bootstrapping process.

The implementation would look like this:

• Gather hardware, networking and maybe partition information.
• Evaluate configuration and build NixOS system.
• Inject a initrd+kernel somewhere in the current system (maybe search for a continuous region within the free disk space).
• Overwrite boot loader with pointers to the injected initrd+kernel.
• Sync
• reboot -f
• Erase disk and repartition (at least in the first implementation of nixos-assimilate, because detecting available partitions and reusing them could become quite difficult).
• Create filesystems and mount.
• Copy closures into the new filesystem layout.
• pivot_root into the new system and bootup NixOS.

In order to properly (re)partition, we need to do the refactoring of nixpart to first of all not be based on kickstart syntax and also have a sane way to specify filesystem layouts.

nixpart is based on blivet, which works with partitioning information in a tree-like structure and also is able to detect existing filesystem layouts, so it's perfectly suited to work with Nix expressions.

That means we need to refactor nixpart to use Nix expressions to specify the filesystem layout and remove all kickstart-related stuff (kickstart is more than just a way to specify partitioning and has dependencies and complexity which we really don't need).

And of course, before we could do that, we need to come up with a sane layout for specifying partitions, possibly something that would blend in seamlessly with the fileSystems attribute set.

This possibly affects/fixes NixOS/nixops#168 and #1080.

[offlinehacker]

Nice :)
On Mar 31, 2014 12:28 AM, "aszlig" [email protected] wrote:

This command would not only be very useful for users to switch to NixOS
but also would get NixOps https://github.com/NixOS/nixops to support a
large variety of different hosters. Also this would simplify the Hetzner
bootstrapping process.

The implementation would look like this:

• Gather hardware, networking and maybe partition information.
• Evaluate configuration and build NixOS system.
• Inject a initrd+kernel somewhere in the current system (maybe search
  for a continuous region within the free disk space).
• Overwrite boot loader with pointers to the injected initrd+kernel.
• Sync
• reboot -f
• Erase disk and repartition (at least in the first implementation of
  nixos-assimilate, because detecting available partitions and reusing
  them could become quite difficult).
• Create filesystems and mount.
• Copy closures into the new filesystem layout.
• pivot_root into the new system and bootup NixOS.

In order to properly (re)partition, we need to do the refactoring of
nixpart https://github.com/aszlig/nixpart to first of all not be based
on kickstart syntax and also have a sane way to specify filesystem layouts.

nixpart is based on blivet https://fedoraproject.org/wiki/Blivet, which
works with partitioning information in a tree-like structure and also is
able to detect existing filesystem layouts, so it's perfectly suited to
work with Nix expressions.

That means we need to refactor nixpart to use Nix expressions to specify
the filesystem layout and remove all kickstart-related stuff (kickstart is
more than just a way to specify partitioning and has dependencies and
complexity which we really don't need).

And of course, before we could do that, we need to come up with a sane
layout for specifying partitions, possibly something that would blend in
seamlessly with the fileSystems attribute set.

This possibly affects/fixes NixOS/nixops#168https://github.com/NixOS/nixops/issues/168and

1080 https://github.com/NixOS/nixpkgs/issues/1080.

Reply to this email directly or view it on GitHubhttps://github.com/NixOS/nixpkgs/issues/2079
.

[shlevy]

+1. Would also be nice to work with pxe and livecd boots too. Do you have an idea of where others might be helpful here?

[offlinehacker]

I would like to help here :)
On Apr 5, 2014 8:31 PM, "Shea Levy" [email protected] wrote:

+1. Would also be nice to work with pxe and livecd boots too. Do you have
an idea of where others might be helpful here?

Reply to this email directly or view it on GitHubhttps://github.com/NixOS/nixpkgs/issues/2079#issuecomment-39646760
.

[aszlig]

Created a wiki at the nixpart repository, which details some basic design about the upcoming version 1.0 which is needed for nixos-assimilate. I'm open for suggestions and/or help on that front (also need to specify the format of the device tree, but that's coming soon), so far there is only the partition remapper, please let me know what you think and/or add possible problems/obstacles to the draft:

https://github.com/aszlig/nixpart/wiki/Partition-remapper-%28draft%29

[lethalman]

Somebody, can't remember if mm_freak or whoelse, installed nixos simply in a subdirectory like /nixos. Then somehow install grub in some device (probably a file in the host file system), and add this device to be chainloaded from the main bootloader. In the hope that grub supports set root=(...)/nixos.
Finally modified initrd from configuration.nix to change root to /nixos.
This is a great thing that should be documented somewhere, or be part of nix-assimilate, in that it keeps your existing partitions and system intact, yet you are able to run nixos natively.
I'm sure a large number of people (like me) would instantly install it.

[aszlig]

That's what I did on some systems as well, and if you have something like btrfs, you could simply change root subvolumes for switching distros. And that's what I had in mind if you want to keep your current partitioning layout. The block device remapper is only if you want to change the partitioning while retaining current data, so essentially we're doing exactly the same but just change the layout (might break the existing system however).

The main reason for the block device remapper is for supporting different dedicated server hosting providers for NixOps. Most of these hosters are preinstalling an image with a fixed filesystem layout, so it's a way to change that without the need to do manual intervention.

[CMCDragonkai]

This is such a cool idea.

Somebody managed to do this for CoreOS if that helps.

https://github.com/ibuildthecloud/coreos-on-do

It manages to "assimilate" DO's VM into a CoreOS instance using kexec.

[wmertens]

Ooh I didn't think of that use case, neat!

Digital Ocean is a very cheap provider so having NixOS on there would be
awesome. I wonder if the kexec is required, a simple reboot should work as
well?

I also took the multiple btrfs subvols approach, and I keep my old Ubuntu
mounted to run not-yet-nixified software in a chroot.
On Aug 21, 2014 7:45 AM, "Roger Qiu" [email protected] wrote:

This is such a cool idea.

Somebody managed to do this for CoreOS if that helps.

https://github.com/ibuildthecloud/coreos-on-do

It manages to "assimilate" DO's VM into a CoreOS instance using kexec.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2079#issuecomment-52880766.

[CMCDragonkai]

I think the coreos solution does a couple reboots in order to make it work.

[CMCDragonkai]

Does blivet work with ZFS? That's what I use. And it would be so cool if could assemble ZFS + LUKS.

[aszlig]

@CMCDragonkai: Unfortunately, it doesn't support ZFS yet :-/

[CMCDragonkai]

I can help with ZFS integration, but you're intending to port blivet to nixpart?

[aszlig]

Well, nixpart is already using blivet, but right now it's based on a quick & dirty kickstart-based implementation.

[CMCDragonkai]

Would be great without the python dependency, straight shell scripts might be nice. I mainly don't want to have useless interpreters on my system that is used only for one thing (I use it for throwaway lightweight servers). A compiled small binary might be acceptable.

[aszlig]

Straight shell scripts are going to be a major headache to maintain, not only do we need to re-implement blivet, but we also will have to handle all the platform-specifics regarding to external commands. I choose Python because there was blivet available (and as Fedora is using this for storage setup, it's already well tested) and it is available on all major distros, if I were free to choose the language, I'd have gone for Haskell.

[copumpkin]

:+1: :+1: :+1: :+1: :+1:

[wmertens]

So in summary, to replace a system with NixOS while retaining the partitioning, these are the minimal steps:

1. Be root
2. Install Nix as root (# curl https://nixos.org/nix/install | sh) and load the profile script.
3. Somehow obtain nixos-generate-config and nixos-install which are in nixos, not nixpkgs
4. Init /nixos: # mkdir /nixos && nixos-generate-config --root /nixos
5. Edit the generated NixOS config:
   
   
   
   1. Tell NixOS that it lives under /nixos
   
   
   2. Make sure to include the correct sshd config and networking
   
   
6. Run nixos-install
7. Reboot, NixOS should come up

Right? I'm wondering about having /nix twice, once as /nix and once as /nixos/nix. Perhaps a bind mount?

Another option would be to tweak the Nix installer so it immediately allows installing into a chroot. So then you'd run a script and you end up in a chrooted NixOS environment, ready to run nixos-rebuild. Probably better? For that the steps would be:

1. As root, run # curl https://nixos.org/nixos/install | sh _(doesn't exist yet)_
2. This creates NixOS chroot:
   
   
   
   1. Under /nixos unless overridden with NIXOS_ROOTDIR=...
   
   
   2. Bind mounts /dev, /proc and /sys
   
   
   3. Auto-generates config with subdir-root already set
   
   
   4. Opens chroot'ed shell.
   
   
   5. Does not run systemd. Is that a problem?
   
   
3. Complete installation by tweaking NixOS config, nixos-rebuild boot and reboot
   
   
   
   1. This can of course be automated on predictable environments like Digital Ocean, perhaps using environment variables to pass the initial config to the NixOS install script so the entire installation is a single SSH command.
   
   

Makes sense? Am I missing something? Note that /home can thus be shared between OSes...

[CMCDragonkai]

Does this replace the existing OS? What about the boot records, NixOS
allows you to change between configuration changes during boot.

On 30/09/2014 11:20 PM, wmertens wrote:

So in summary, to replace a system with NixOS while retaining the
partitioning, these are the minimal steps:

1. Be root
2. Install Nix as root (|# curl https://nixos.org/nix/install | sh|)
   and load the profile script.
3. Somehow obtain |nixos-generate-config| and |nixos-install| which
   are in nixos, not nixpkgs
4. Init |/nixos|: |# mkdir /nixos && nixos-generate-config --root
   /nixos|
5. Edit the generated NixOS config:
   
   
   
   1. Tell NixOS that it lives under |/nixos|
   
   
   2. Make sure to include the correct sshd config and networking
   
   
6. Run |nixos-install|
7. Reboot, NixOS should come up

Right? I'm wondering about having |/nix| twice, once as |/nix| and
once as |/nixos/nix|. Perhaps a bind mount?

Another option would be to tweak the Nix installer so it immediately
allows installing into a chroot. So then you'd run a script and you
end up in a chrooted NixOS environment, ready to run nixos-rebuild.
Probably better? For that the steps would be:

1. As root, run |# curl https://nixos.org/nixos/install | sh|
   /(doesn't exist yet)/
2. This creates NixOS chroot:
   
   
   
   1. Under |/nixos| unless overridden with |NIXOS_ROOTDIR=...|
   
   
   2. Bind mounts |/dev|, |/proc| and |/sys|
   
   
   3. Auto-generates config with subdir-root already set
   
   
   4. Opens chroot'ed shell.
   
   
   5. _Does not_ run |systemd|. Is that a problem?
   
   
3. Complete installation by tweaking NixOS config, |nixos-rebuild
   boot| and |reboot|
   
   
   
   1. This can of course be automated on predictable environments
      
      like Digital Ocean, perhaps using environment variables to
      
      pass the initial config to the NixOS install script so the
      
      entire installation is a single SSH command.
   
   

Makes sense? Am I missing something? Note that |/home| can thus be
shared between OSes...

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2079#issuecomment-57312059.

Founder of Polycademy & SnapSearch
http://polycademy.com
https://snapsearch.io
[phone]

[aszlig]

@wmertens: nixos-generate-config and nixos-install are in <nixpkgs> as well, so you can build them on a non-NixOS system. Also, I wouldn't go the nixos-rebuild route, because we essentially just need a store path to the new NixOS system, so something like nix-build -E 'import <nixpkgs/nixos> { configuration = ./foo/bar; }.system' should be sufficient. For retaining the partition layout, it's enough to just write a boot loader(-entry) to point to the kernel/initrd store paths and move away the old system in postMountCommands.

@CMCDragonkai: The long-term goal is to have a way to rollback to the existing OS in case something goes wrong, maybe by tar'ing (because we want to retain file permissions) everything except the kernel and modified initrd it into a store path so that it can easily be garbage-collected. But for the first implementation I'm aiming for destructive operation only, essentially wiping out the current system and create a new partition layout.

[CMCDragonkai]

That makes sense. I'm only interested in destructive updating, because
some cloud providers like DO don't provide the ability to run custom
images. So I'm guessing when this process is ready, I can just run a
program and change an Ubuntu into a NixOS?

On 1/10/2014 9:01 AM, aszlig wrote:

@wmertens https://github.com/wmertens: |nixos-generate-config| and
|nixos-install| are in || as well, so you can build them on a
non-NixOS system. Also, I wouldn't go the |nixos-rebuild| route,
because we essentially just need a store path to the new NixOS system,
so something like |nix-build -E 'import {
configuration = ./foo/bar; }.system'| should be sufficient. For
retaining the partition layout, it's enough to just write a boot
loader(-entry) to point to the kernel/initrd store paths and move away
the old system in |postMountCommands|.

@CMCDragonkai https://github.com/CMCDragonkai: The long-term goal is
to have a way to rollback to the existing OS in case something goes
wrong, maybe by tar'ing (because we want to retain file permissions)
everything except the kernel and modified initrd it into a store path
so that it can easily be garbage-collected. But for the first
implementation I'm aiming for destructive operation only, essentially
wiping out the current system and create a new partition layout.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/2079#issuecomment-57396357.

Founder of Polycademy & SnapSearch
http://polycademy.com
https://snapsearch.io
[phone]

[aszlig]

@CMCDragonkai: Essentially yes, that's what it's all about :-)

[Fuuzetsu]

Very cool, I'm also a victim of Digital Ocean having very poor other-than-default distro support.

[aszlig]

Added a wiki page for the device tree representation format. My current WIP implementation is based on variant 1, but I'm not quite happy with it (it feels so verbose), so I'm open for better suggestions on how to represent the device tree in a Nix-friendly way.

[lethalman]

On 08/10/2014 14:01, aszlig wrote:

Added a wiki page for the device tree representation format
https://github.com/aszlig/nixpart/wiki/Device-tree-representation-%28draft%29.
My current WIP implementation is based on variant 1, but I'm not quite
happy with it (it feels so verbose), so I'm open for better
suggestions on how to represent the device tree in a Nix-friendly way.

Don't use a tree in nix at all. Use systemd dependencies. That's what
I'm doing with systemd-initrd in my branch, in fact, to allow complex
dependencies in initrd and remove the preLVM stuff.

[aszlig]

@lethalman: I'm not using a "tree of attributes" on the Nix part, but we already discussed this on IRC and I've added a variation that's similar to systemd units, thanks for the suggestion.

So far I guess we pretty much ruled out variation 1 and 2 and it's probably going to be either 3, 4 or something in between.

[hsenag]

Does this actually have an implementation yet? I can't spot anything in this thread, but it seems a bit optimistic to get it into 14.11 if it's still just an idea.

[aszlig]

@hsenag: I'm trying to get nixpart 1.0 finished within the next few days, which is about 50% (or at least I _hope_ it's 50%) of nixos-assimilate. I know it's kinda optimistic but trying to finish it for 14.11 at least keeps me pragmatic and it's not a release-critical thing.

[domenkozar]

@aszlig I'll bump this to 15.05, let me know if it's doable.

[CestDiego]

What is the status of this? Thank you guys for everything you do for the future of NixOps

[thoughtpolice]

Definitely not getting done for this release; punting

[aszlig]

Submitted a talk about it for NixCon 2015, so I'm going to work on this during October.

[jeaye]

This would keep me on DO, instead of forcing me onto Linode or some other providers. One key issue with that, however, is that DO makes some assumptions about how the filesystem is laid out. I made an issue on digitalocean-debian-to-arch about this, which was ultimately worked out.

Either way, if other NixOS folks are trying this on DO, this will almost certainly become an issue.

[wmertens]

It seems to me that a 'light' version which throws away the current linux entirely and installs nixos with a desired configuration on the current partition layout is also desirable. Steps:

1. Get root access to system
2. Install temporary nixos under /nixosconvert. It should also get the desired configuration, which should fit the current partition layout and system.
3. Install nixos boot loader that bind mounts /nixosconvert to /
4. Reboot
5. Nixos converter system boots, mounts original root, clears it out except desired directories, and installs nixos on it. Boot loader gets overwritten with proper configuration
6. Reboot, welcome to your new NixOS.

I believe there was support for bind mounting in the initrd but it was taken out, not sure.

Other than that, this needs the nixos installer image (doesn't even need to be nixos itself) and a service that runs the install with the desired configuration, rebooting afterwards.

Thoughts?

[jeaye]

@wmertens I have developed such a "light" version which is even simpler than that! Have a look at nixos-in-place and my relevant post to the nix-dev list.

[wmertens]

@jeaye nice! In a perfect world it would install nixos without the ISO download, via a nix install which evaluates system, but this also works :)

[domenkozar]

@aszlig what's the status on this? Should we bump the milestone?

[elitak]

nixos-in-place wasn't working for me on Digital Ocean, so I made an even simpler version that you can pipe straight to bash over ssh, and uses the nix install script to bootstrap, rather than the ISO: https://gist.github.com/elitak/996e3b3c54c07409c8f2 . I'm next going to look into integrating the DO API into nixops to just instantiate a droplet and run my script to transform it to nixos.

I look forward to the final nixpart + dm-module solution, though; that should be pretty useful for installing on every platform.

[jeaye]

@elitak It's a shame to see yet another project trying to do this. It'd be great if you applied your spare time to working on nixos-assimilate or nixos-in-place instead!

[elitak]

@jeaye I hesitate to call it a "project". It's essentially a couple dozen lines of bash that I intend to discard once nixos-assimilate is ready. I don't consider it shameful to try to help everyone expedite their deployments by giving them something simple that works. I needed to write that script for my own use, whether I shared it or not.

I would like to contribute to nixos-assimilate, but it's pretty hard when none of it is shared. I can't find any work on the dm-module @aszlig mentioned in his nixcon talk, nor the nixpart changes. I checked aszlig/nixpkgs and aszlig/nixpart. If you know where the most recent work is published, please point me to it and I'd be more than happy to look and see what I can offer. Otherwise, I just have to wait until help is being accepted. I take the absence of updates here and in the repositories as an indication it isn't.

As for nixos-in-place, it didn't work as advertised to begin with (see notes in the comments of my script), so that route started with extra obstacles in it.

[aszlig]

Having nixos-in-place isn't really duplicate work, because it's working with the existing filesystem IIUC, while nixos-assimilate goal is to do repartitioning on the fly.

As for the published work, I'm currently focusing on nixpart and trying to rebase the nixpart branch on top of current master with the new upstream version(s). Unfortunately there are a few obstacles with the tests of blivet and libblockdev, because apparently it seems that not even the upstream project runs its own test suite :-/

I'll try to get the nixpart branch in nixpkgs finished over the next two weeks, so we should at least have the necessary dependencies ready to go with nixpart. The reason why I didn't yet push the new version of nixpart is that blivets API was constantly changing at the time I started to work on it, so it's currently a bunch of experiments (part of those I have shown at the talk).

[elitak]

@aszlig I appreciate the update. I will have a cursory look at those test suites to see if there's anything easy to fix.

Where is the device-mapper module that will do the inode/block tagging? Is that part of libblockdev or is it still unwritten?

I still have a very nebulous picture of how it will all fit together. A repository tying them all together would be helpful. To that end, where can I find nixpart commits newer than 3 years ago?

If the answers to most of these questions are going to be self-evident in two weeks or so, I'm fine with waiting. I don't want to distract you from actual progress when its unlikely I'll be able to contribute without the groundwork you're doing anyway.

Thanks for your efforts on what I consider a real killer app for NixOS.

[0xABAB]

@elitak cat customConfig.optional nixos-infect | ssh root@targethost bash is a fragile construct. If you make it such that you first copy the code to the host and then execute it in a remote screen for example, it wouldn't be. I didn't look further than that, but I am happy to see people moving in this direction.

[wmertens]

@elitak so you're saying that the changes introduced by https://github.com/NixOS/nixpkgs/commit/171d43ba4f51384670c8af0e5188c023f5bb0d94 are too brittle on other distros?

Can you perhaps try with nixos-install that does not have the above patch? Copying all those files around doesn't seem very efficient :(

[elitak]

I didn't realize gists didn't have issue tracking, so I've now promoted the gist to a project:
https://github.com/elitak/nixos-infect
Please file issues there so we keep comments here pertinent to nixos-assimilate.

@wmertens I only saw the "unshare" problem on Gentoo ARM hosts when I was performing the installation manually. The issue usually preventing bind/no-mount roots on cloud VPSes is the bash permission error one:
https://github.com/NixOS/nixpkgs/issues/10230

The installation needs to perform at least 2 copies anyway: one from the running system to the newroot, and one rsync back. The only cost incurred by that bug is that you need to zero out about 2GB for a loopback image. It's possible to do everything at once inside nixos-install, but that's out of the scope of my script. If I need dozens of NixOS VPSes, I will clone the first to the subsequent ones using either snapshots/imaging provided by the vendor or rsync -a --delete /tmp/rootbindmount/ secondhost:/tmp/rootbindmount/ (then reboot -f secondhost). I don't want to be putting that much load on the NixOS binary cache servers anyway; cloud-local rsync/snapshotting is the way to go.

@0xABAB, I know it's fragile; it's up to the user to know for himself if he needs something better. I'm just recommending the simplest, most optimistic way to do it. That optimism is a theme throughout the script. I want to keep it as simple as possible so that fixes can be easily made by users. The script is not idempotent, so the only thing screen really adds is avoidance of termination-by-disconnect.

nixos-assimilate, whenever it drops, will make the effort to handle every case. I will contribute to that endeavor as soon as I'm allowed to, starting with Digital Ocean support.

[wmertens]

@elitak why do you need to rsync back, why not simply move or bind mount root on a subdir?

[obadz]

I've got a PR for something related but a lot less ambitious (no partition layout changes) if anyone is interested: #17784

[danbst]

@aszlig are there plans to publish the unfinished nixpart 1.0?

[aszlig]

@danbst: Not really, because I need to rewrite a lot of this stuff for latest blivet. But I could publish the unfinished packaging of blivet, because that's the biggest stopper last time I was at it and it would help to have it packaged (along with full NixOS tests).

[aszlig]

I just pushed the rebased nixpart branch along with working tests of libblockdev. Finishing the blivet tests ASAP (after sleep + work) and going to rewrite nixpart from scratch, because the blivet API changed a lot since the last time I was working on it. The upside is that the API has changed in a way that the actual implementation of nixpart should be only a matter of a few lines of code.

[aszlig]

See #21403.

[gilligan]

@aszlig i suppose this is still in progress somehow?

[roosemberth]

What's the state of this PR?

[danbst]

@gilligan @roosemberth
I'll answer for @aszlig here, but he may correct me

This PR is blocked by https://github.com/NixOS/nixpkgs/pull/21403, without it being merged nixos-assimilate isn't much better than nixos-infect, nixos-in-place and co.

The nixpart PR itself is simply not finished. IMO it would be great to split kickstart/blivet part of it into separate PR and merge faster. It would then allow to kexec a running system into NixOS-in-RAM, do the partitioning (via kickstart) and install using normal way + some network initialization magic from nixos-infect. (https://nixos.wiki/wiki/Install_NixOS_on_a_Server_With_a_Different_Filesystem)

[petabyteboy]

I think it's not too hard to realize something like this using kexec. I am using @cleverca22's kexec files to automatically install NixOS to Hetzner Cloud VMs, though it is not integrated with NixOps.
Basically I just took the kexec script and added a service that runs the necessary commands to install NixOS, and reboots.

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[aszlig]

still important to me