I'm about to get some NixOS stuff setup in lxd so I'll try and document
that as I go along. It wasn't hard to do libvirt-lxc last time I worked
with it.

On Tue, Sep 8, 2015, 16:59 Rohit Yadav [email protected] wrote:

I am not sure if this has been answered before. Does anyone know as how to
run NixOS inside LXC container on Ubuntu 14.04 as host. This would be
really helpful for me to test all my ideas and then run NixOS on host
machine. I tried searching for this online but could not find any help.
lxc-create command on ubuntu only provides limited set of OS as option. I
really dunno know how to create manually rootfs which can then be booted.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/issues/9735.

wkennington on 9 Sep 2015

It would be great if you could create some documentation for the same. Could you please point me to some references to create bootable rootfs image.

rowhit on 9 Sep 2015

(triage) Any new leads?

Profpatsch on 23 Jul 2016

I tried this yesterday and it's working now. Since I couldn't find a guide for this I just did this in an ad-hoc kind of way:

Step by step guide

Install VirtualBox and boot up the NixOS installer ISO (for this guide I'm going to assume the name to be NixOS)
Install NixOS into VirtualBox following the official manual (requires about 1.5 GiB VDI disk space)
Optionally test NixOS in VirtualBox, then stop the VM
Open the directory of the VDI in a terminal and run: VBoxManage internalcommands converttoraw NixOS.vdi NixOS.img
(this will convert the VDI into a raw disk image that can be mounted) [[source](https://geekwentfreak-raviteja.rhcloud.com/blog/2011/08/09/mount-virtualbox-image-vdi-on-linux/)]
Set up a partitioned loop back disk using the raw disk image: sudo losetup -P /dev/loop0 NixOS.img
Mount the first partition of the loopback image: sudo mount /dev/loop0p1 /mnt
Create an empty LXC image: sudo lxc-create --name nixos --template=none [[source](https://snikt.net/blog/2014/03/22/convert-kvm-image-to-lxc-container/)]
Copy the contents of the NixOS installation into the container root filesystem directory: sudo rsync -av /mnt/* /var/lib/lxc/nixos/rootfs/
Edit the container configuration in _/var/lib/lxc/nixos/config_ and at least set
(without it won't properly start, complete example file below)
- lxc.aa_allow_incomplete = 1
- lxc.init_cmd = /nix/var/nix/profiles/system/init
- lxc.mount = /var/lib/lxc/nixos/fstab
Create the file _/var/lib/lxc/nixos/fstab_ and add at least this line to work around some weird issue:
- tmpfs run tmpfs rw,nodev,size=255648k,mode=755 0 0
Boot up the container: lxc-start -F -n nixos
Inject root shell into container: lxc-attach -n nixos
In the shell:
1. Edit _/etc/nixos/configuration.nix_ using nano: nano /etc/nixos/configuration.nix
  - Enable container mode: boot.isContainer = true;
  - Make sure GRUB is disabled: boot.loader.grub.enable = false;
2. Apply new configuration: nixos-rebuild switch
  - If network access doesn't work run this one-time hack to get networking to work before trying again:
    
    systemctl stop dhcpcd && dhcpcd eth0

Suggestions on how to make this shorter (particularly on how to not having to use a VM at all) are appreciated.

Example container configuration

# Template used to create this container: /usr/share/lxc/templates/lxc-debian
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.hwaddr = 00:00:00:00:00:00 # CHANGE ME
# Common configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.aa_allow_incomplete = 1
# Container specific configuration
lxc.arch = amd64
lxc.rootfs = /var/lib/lxc/nixos/rootfs
lxc.utsname = nixos
lxc.mount = /var/lib/lxc/nixos/fstab
lxc.init_cmd = /nix/var/nix/profiles/system/init

ntninja on 21 Aug 2016

👍2

Since this commit, I think, the LXC container might be unusable. The symptoms will be that once attached (through lxc-attach), no commands are available, /run/current-system being nowhere to be found.

A fix that works, is to let LXC mount /run, which would normally be mounted by the stage1 script. (The LXC container starts at stage2.) Adding a lxc.mount.entry config is fine, otherwise adding to the fstab used by lxc.mount would work too.

lxc.mount.entry = tmpfs run     tmpfs   rw,nodev,size=255648k,mode=755  0       0

Be sure to use run and not /run or else LXC will ignore the mountpoint.

Otherwise, it _seems_ everything else provided by @alexander255 works. Thanks @alexander255 !

samueldr on 19 Nov 2016

We should provide a LXC template script to be able to create NixOS LXC containers more easily. Moreover, the settings needed for LXC should be documented in the NixOS manual.

fpletz on 22 Nov 2016

👍11

I tried following your instructions @alexander255 on Arch Linux but I get this when booting the container with lxc-start with the --logfile option:

      lxc-start 20170305011300.554 ERROR    lxc_cgfsng - cgroups/cgfsng.c:create_path_for_hierarchy:1306 - Path "/sys/fs/cgroup/systemd//lxc/cuauv" already existed.
      lxc-start 20170305011300.554 ERROR    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_create:1363 - No such file or directory - Failed to create /sys/fs/cgroup/systemd//lxc/cuauv: No such file or directory
      lxc-start 20170305011300.559 ERROR    lxc_conf - conf.c:setup_mount:2001 - No such file or directory - failed to use '/var/lib/lxc/cuauv/fstab'
      lxc-start 20170305011300.559 ERROR    lxc_conf - conf.c:lxc_setup:3864 - failed to setup the mounts for 'cuauv'
      lxc-start 20170305011300.559 ERROR    lxc_start - start.c:do_start:811 - Failed to setup container "cuauv".
      lxc-start 20170305011300.559 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 3)
      lxc-start 20170305011300.559 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "cuauv".
      lxc-start 20170305011300.598 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
      lxc-start 20170305011300.598 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.

alexozer on 5 Mar 2017

@alexozer: Try running sudo touch /var/lib/lxc/cuauv/fstab to create the file system table file, then try again.

ntninja on 5 Mar 2017

@samueldr I've updated the guide based on your feedback. Thanks!

ntninja on 5 Mar 2017

@alexander255 Adding the entry to fstab (rather than the lxc config) seems to fix everything for me, thanks! (Although maybe I did something wrong before and adding to lxc config would work...)

alexozer on 5 Mar 2017

I've been running a container from the nixpkgs containerTarball. It's simpler than the procedure above, and has been working well for me.

This procedure requires a working nix package manager. It would be nice to have the containerTarball available somewhere to eliminate this dependency.

Get image and extract it to LXC

# Generate the image on a machine with nix.
nix-build -E '(import <nixpkgs/nixos/release.nix> {}).containerTarball.x86_64-linux'

# On the Ubuntu machine:
apt install lxc1
lxc-create -n my-nixos -t /bin/true # /bin/true creates empty rootfs
cd /var/lib/lxc/mynixos/rootfs/
tar xf /path/to/nixos-system-x86_64-linux.tar.xz

LXC Configuration example

# [/var/lib/lxc/my-nixos/config]
# These are the defaults from Ubuntu.
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:f6:ff:de
lxc.rootfs = /var/lib/lxc/my-nixos/rootfs
lxc.rootfs.backend = dir

# Added for NixOS.
lxc.include = /usr/share/lxc/config/common.conf
lxc.init_cmd = /init

Boot

lxc-start -n my-nixos # If you need debugging: --logfile /tmp/lxc.log --logpriority info
lxc-attach -n my-nixos # Get a shell

orbekk on 13 May 2017

👍10

Prebuilt versions of such container tarballs can be found on Hydra:

pacien on 30 Jan 2018

Hello, I'm trying to run https://hydra.nixos.org/job/nixos/release-17.09/nixos.containerTarball.x86_64-linux into and user-space lxc1 container. I get errors like:

/init: line 9: /proc/cmdline: No such file or directory

<<< NixOS Stage 2 >>>

mount: /dev: permission denied.
mount: /proc: permission denied.
findmnt: can't read /proc/mounts: No such file or directory
/init: line 118: /dev/fd/62: No such file or directory
running activation script...
tee: /proc/self/fd/11: No such file or directory
tee: /proc/self/fd/11: No such file or directory
setting up /etc...
mount: /dev: permission denied.
mount: /dev/pts: permission denied.
mount: /dev/shm: permission denied.
mount: /proc: permission denied.
mount: /run: permission denied.
mount: /run/keys: permission denied.
mount: /run/wrappers: permission denied.
NOTE: Under Linux, effective file capabilities must either be empty, or
      exactly match the union of selected permitted and inheritable bits.
Failed to set capabilities on file `/run/wrappers/wrappers.H4A6RGttrG/ping' (Operation not permitted)
error: changing ownership of path ‘/nix/store’: Operation not permitted
GC Warning: pthread_getattr_np or pthread_attr_getstack failed for main thread
GC Warning: Couldn't read /proc/stat
error: changing ownership of path ‘/nix/store’: Operation not permitted
starting systemd...
Failed to insert module 'autofs4': No such file or directory
Failed to insert module 'unix': No such file or directory
Failed to mount sysfs at /sys: Operation not permitted
Failed to mount proc at /proc: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

Is this supposed to work ?

My config is:

# Template used to create this container: /bin/true
# Parameters passed to the template:
# Template script checksum (SHA-1): 61ac3e136ac6e6134b56e54350899e34bfb84713
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.apparmor.allow_incomplete = 1

lxc.rootfs.path = dir:/home/hawk/.local/share/lxc/nixos/rootfs

lxc.init.cmd = /init
# lxc.include = /usr/share/lxc/config/common.conf

Note I had to comment out the last line because of :

lxc-start: nixos: cgroups/cgfsng.c: lxc_cgroup_set_data: 2039 Failed to setup limits for the "devices" controller. The controller seems to be unused by "cgfsng" cgroup driver or not enabled on the cgroup hierarchy
                        lxc-start: nixos: cgroups/cgfsng.c: cgfsng_setup_limits: 2076 Success - Error setting devices.deny to a for nixos
                                                                                                                                         lxc-start: nixos: start.c: lxc_spawn: 1345 Failed to setup the devices cgroup for container "nixos".
                                                lxc-start: nixos: start.c: __lxc_start: 1459 Failed to spawn container "nixos".
                                                                                                                               lxc-start: nixos: tools/lxc_start.c: main: 371 The container failed to start.
lxc-start: nixos: tools/lxc_start.c: main: 375 Additional information can be obtained by setting the --logfile and --logpriority options.

hawk78 on 30 Jan 2018

I have tested nixos with LXD 2.21 recently and it runs really well. To create nixos container do the following:

1) create rootfs image from any system running nixOS:

nix-build -E '(import <nixpkgs/nixos/release.nix> {}).containerTarball.x86_64-linux'

2) create lxd image metadata:

templates/hostname.tpl:

{{ container.name }}

metadata.yaml:

architecture: "x86_64"
creation_date: 1519291500
properties:
    architecture: "x86_64"
    description: "NixOS 17.09"
    os: "nixos"
    release: "17.09"
templates:
    /etc/hostname:
        when:
            - create
            - copy
        template: hostname.tpl

3) create image and rootfs tarbals:

tar -zcvf ../nixos-system-x86_64-linux.tar.gz *
xzcat ../nixos-system-x86_64-linux.tar.xz | gzip > ../nixos-system-x86_64-linux.tar.gz.root

4) import image to lxd:

lxc image import ./nixos-system-x86_64-linux.tar.gz ./nixos-system-x86_64-linux.tar.gz.root --alias nixos-17.09

5) create default-nix LXD profile

config:
  raw.lxc: |-
    lxc.init_cmd = /init
    lxc.aa_allow_incomplete = 1
description: Default LXD profile for NixOS
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default-nix

6) create your first NixOS container and have fun:

lxc launch -p default-nix nixos-17.09 nixos-first
lxc exec nixos-first -- /bin/sh -l

rejsmont on 6 Mar 2018

👍4 ❤1

Thank you for the proposed solution. It is working. Now I have a question. The container is raised, you can connect to it by lxc exec nixos-first -- /bin/sh -l . But you can not connect to it by ssh. And there is no network inside.
How to make this container suitable for use with nixops?

ivanovs-4 on 23 Mar 2018

Hi @ivanovs-4,

I do not have this issue. DHCP gets up after a while (10-60sec). For ssh to work, you need to enable it in your /etc/nixos/configuration.nix it. But why would you need ssh when you have lxc exec? Unless you want to use rsync, etc...

Test your network with other lxd containers to make sure it works.

I have not used nixops, so cannot help you with the second question.

rejsmont on 23 Mar 2018

Hi @rejsmont. Thank you for your answer.

I checked the network with the container ubuntu. It works right away. The container gets an ipv4 address.
The nixos container gets an ip6 address. But the Internet is not accessible through it. "Could not resolve host name".
I event can't restart dhcp:

[root@nixos:~]# systemctl stop dhcpcd
Failed to connect to bus: No such file or directory

I guess I need to make a more fully functional image of the system instead of the one that the command creates nix-build -E '(import <nixpkgs/nixos/release.nix> {}).containerTarball.x86_64-linux'.
Do not know how.

ivanovs-4 on 23 Mar 2018

Hello,

I downloaded the 32 bit prebuilt rootfs from hydra and managed to create the container following the instructions above. The container booted fine :smiley:

I edited the /etc/nixos/configuration.nix to customize my container and tried to rebuild the system:

$ nixos-rebuild switch
Error in tempdir() using /run/user/0/nix-build.XXXXXX: Parent directory (/run/user/0) does not exist at /nix/store/ifkz8p3p6cy1x6fnyks8nfy64ibnvais-nix-1.11.16/lib/perl5/site_perl/5.24.3/i686-linux-thread-multi/Nix/Utils.pm line 45

Creating the missing directory manually

mkdir /run/user/0

makes the error message disappear and the system works as expected. However, it seems now after every reboot I have to create this missing directory by hand.

wagdav on 31 Mar 2018

I tried to run NixOs 18.09 on lxd 3.8 with the setup above. I'm able to create the container but cannot execute commands like nix-channel --update or nixos-rebuild switch.

[root@nixos:~]# nix-channel --update
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: while setting up the build environment: mounting /proc: Operation not permitted
error: program '/nix/store/rffcxk0l94lc96yl07r26sdnfql6x0h4-nix-2.1.3/bin/nix-env' failed with exit code 1

(first call)

[root@nixos:~]# nix-channel --update
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: while setting up the build environment: mounting /proc: Operation not permitted
error: program '/nix/store/rffcxk0l94lc96yl07r26sdnfql6x0h4-nix-2.1.3/bin/nix-env' failed with exit code 1

[root@nixos:~]# nixos-rebuild switch
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:13
building Nix...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
these paths will be fetched (15.16 MiB download, 65.01 MiB unpacked):
  /nix/store/12zhmzzhrwszdc8q3fwgifpwjkwi3mzc-gcc-7.3.0-lib
  /nix/store/1nlj1w7pqmxvxw9wi98ssngab9fsm091-libkrb5-1.15.2
  /nix/store/35yalbjrb5jh0arjawfxkryppvpip11a-nghttp2-1.24.0-lib
  /nix/store/3cx8jscamhp08hjv6x4lhd1w57ggji0k-gzip-1.9
  /nix/store/60pn8jnvbb3n70337jvdfkdn2zx7f331-libsodium-1.0.16
  /nix/store/6953iif8k3550lspbrss63w9saprqq8y-busybox-1.28.1
  /nix/store/6k794ln3ila2457d9k9fgyyyq7px8aci-acl-2.2.52
  /nix/store/6p06bw9hggh8vj093a59nib283cs1kbx-libssh2-1.8.0
  /nix/store/6v0aqcar0j6n5mlkdhqid4drqwvhlszn-curl-7.59.0
  /nix/store/7zam9d16ml78rb0p1p5gds4p5z99fpyv-gnutar-1.30
  /nix/store/80gpwb4fkvd2rfrpgmlgsq4hjbsz225p-bzip2-1.0.6.0.1
  /nix/store/8c30dl4mmgm4rip1l08fvad9gca5sk2j-zlib-1.2.11
  /nix/store/8zs94fwrc2mspl9g1ifmyh2gl78nl6nm-aws-sdk-cpp-1.3.22
  /nix/store/920k63py2w97xpfyb5ps9l6wbidvzfjz-coreutils-8.29
  /nix/store/95yylk9akpg65r32a1sq9qhia4bxzzz2-attr-2.4.47
  /nix/store/9f4kav8sa4qf3clc61h06ggyvhi2m728-xz-5.2.3-bin
  /nix/store/9pma2rib627l4iq67j6jl32mxq1mrzr9-sqlite-3.23.1
  /nix/store/a6bzl1xmhrgdyafdg9b1gmvjcf9pk16c-keyutils-1.5.10-lib
  /nix/store/cdcia67siabmj6li7vyffgv2cry86fq8-nix-2.1.3
  /nix/store/dgikkj4r0j0158013byz27s8dp1z9xhd-brotli-1.0.3-lib
  /nix/store/fzsf72mfsy40l5xbyyrkwb0qjdz8jpl0-boehm-gc-7.6.4
  /nix/store/hqspf7pmaz9s8hjjh3agjdmsn0pdj9xp-bzip2-1.0.6.0.1-bin
  /nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131
  /nix/store/ldqd4ynsmxz7nsmlx36bap9z96v3vk74-libatomic_ops-7.6.2
  /nix/store/lw7xaqhakk0i1c631m3cvac3x4lc5gr5-bash-4.4-p12
  /nix/store/pflp65rydzd4wlywgm39mncw6g9v5cvf-libseccomp-2.3.3
  /nix/store/wqn1vcbm0za87cgxdy86p1ar1risgypr-openssl-1.0.2p
  /nix/store/x1rp4zikbiwjw2x7rvzlny4sj4yc99qw-xz-5.2.3
copying path '/nix/store/6953iif8k3550lspbrss63w9saprqq8y-busybox-1.28.1' from 'https://cache.nixos.org'...
copying path '/nix/store/hwwqshlmazzjzj7yhrkyjydxamvvkfd3-glibc-2.26-131' from 'https://cache.nixos.org'...
copying path '/nix/store/ldqd4ynsmxz7nsmlx36bap9z96v3vk74-libatomic_ops-7.6.2' from 'https://cache.nixos.org'...
copying path '/nix/store/95yylk9akpg65r32a1sq9qhia4bxzzz2-attr-2.4.47' from 'https://cache.nixos.org'...
copying path '/nix/store/lw7xaqhakk0i1c631m3cvac3x4lc5gr5-bash-4.4-p12' from 'https://cache.nixos.org'...
copying path '/nix/store/6k794ln3ila2457d9k9fgyyyq7px8aci-acl-2.2.52' from 'https://cache.nixos.org'...
copying path '/nix/store/dgikkj4r0j0158013byz27s8dp1z9xhd-brotli-1.0.3-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/80gpwb4fkvd2rfrpgmlgsq4hjbsz225p-bzip2-1.0.6.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/920k63py2w97xpfyb5ps9l6wbidvzfjz-coreutils-8.29' from 'https://cache.nixos.org'...
copying path '/nix/store/hqspf7pmaz9s8hjjh3agjdmsn0pdj9xp-bzip2-1.0.6.0.1-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/12zhmzzhrwszdc8q3fwgifpwjkwi3mzc-gcc-7.3.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/7zam9d16ml78rb0p1p5gds4p5z99fpyv-gnutar-1.30' from 'https://cache.nixos.org'...
copying path '/nix/store/fzsf72mfsy40l5xbyyrkwb0qjdz8jpl0-boehm-gc-7.6.4' from 'https://cache.nixos.org'...
copying path '/nix/store/3cx8jscamhp08hjv6x4lhd1w57ggji0k-gzip-1.9' from 'https://cache.nixos.org'...
copying path '/nix/store/a6bzl1xmhrgdyafdg9b1gmvjcf9pk16c-keyutils-1.5.10-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/pflp65rydzd4wlywgm39mncw6g9v5cvf-libseccomp-2.3.3' from 'https://cache.nixos.org'...
copying path '/nix/store/1nlj1w7pqmxvxw9wi98ssngab9fsm091-libkrb5-1.15.2' from 'https://cache.nixos.org'...
copying path '/nix/store/60pn8jnvbb3n70337jvdfkdn2zx7f331-libsodium-1.0.16' from 'https://cache.nixos.org'...
copying path '/nix/store/35yalbjrb5jh0arjawfxkryppvpip11a-nghttp2-1.24.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/wqn1vcbm0za87cgxdy86p1ar1risgypr-openssl-1.0.2p' from 'https://cache.nixos.org'...
copying path '/nix/store/9pma2rib627l4iq67j6jl32mxq1mrzr9-sqlite-3.23.1' from 'https://cache.nixos.org'...
copying path '/nix/store/x1rp4zikbiwjw2x7rvzlny4sj4yc99qw-xz-5.2.3' from 'https://cache.nixos.org'...
copying path '/nix/store/8c30dl4mmgm4rip1l08fvad9gca5sk2j-zlib-1.2.11' from 'https://cache.nixos.org'...
copying path '/nix/store/9f4kav8sa4qf3clc61h06ggyvhi2m728-xz-5.2.3-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/6p06bw9hggh8vj093a59nib283cs1kbx-libssh2-1.8.0' from 'https://cache.nixos.org'...
copying path '/nix/store/6v0aqcar0j6n5mlkdhqid4drqwvhlszn-curl-7.59.0' from 'https://cache.nixos.org'...
copying path '/nix/store/8zs94fwrc2mspl9g1ifmyh2gl78nl6nm-aws-sdk-cpp-1.3.22' from 'https://cache.nixos.org'...
copying path '/nix/store/cdcia67siabmj6li7vyffgv2cry86fq8-nix-2.1.3' from 'https://cache.nixos.org'...
/tmp/nixos-rebuild.Qf67Dx/nix
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
building the system configuration...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)

(second call)

[root@nixos:~]# nixos-rebuild switch
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:13
building Nix...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
/tmp/nixos-rebuild.b6c8im/nix
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
building the system configuration...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)

Just a note: Since some lxd versions the raw.lxc command changed. This is working for me:

config:
  raw.lxc: |-
    lxc.init.cmd = /init
    lxc.apparmor.allow_incomplete = 1
description: Default LXD profile for NixOS
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: nixos

asbachb on 2 Jan 2019

👍1

I tried running the steps mentioned above by @rejsmont. I have made the appropriate change to the config as well:

```lxc profile show default-nix
config:
raw.lxc: |-
lxc.init.cmd = /init
lxc.apparmor.allow_incomplete = 1
description: Default LXD profile for NixOS
devices:
eth0:
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default-nix
used_by:

/1.0/containers/nixos-first

But still I receive the following error:
```lxc info --show-log local:nixos-first
Name: nixos-first
Remote: unix://
Architecture: x86_64
Created: 2019/01/04 00:24 UTC
Status: Stopped
Type: persistent
Profiles: default-nix

Log:

lxc nixos-first 20190104002421.441 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 1                                                                       
lxc nixos-first 20190104002421.441 ERROR    start - start.c:lxc_init:859 - Failed to run lxc.hook.pre-start for container "nixos-first"                                      
lxc nixos-first 20190104002421.442 ERROR    start - start.c:__lxc_start:1905 - Failed to initialize container "nixos-first"                                                  
lxc nixos-first 20190104002421.442 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:833 - No such file or directory - Failed to receive the container state

rowhit on 4 Jan 2019

@rowhit I guess I had a similar issue. Can you double check
a) That your metadata.yaml refers the correct folder e.g. template vs. templates
b) The metadata tar contains all files. metadata.yaml and templates/hostname.tpl

asbachb on 4 Jan 2019

@asbachb: Thanks a lot, it worked. I had a typo. I am able to generate the image and run it with updated default-nix config. However, there are still a few minor things to be done to get the system into a working state.

Source the system profile

source /nix/var/nix/profiles/system/etc/profile

Create proc dir (had few issues related to the proc not able to mount).

mkdir /mnt/proc
mount -t proc proc /mnt/proc

Download the nixpkgs. As it cannot find the nixpkgs channel

# on host
wget https://github.com/NixOS/nixpkgs/archive/18.09.tar.gz
lxc file push nixpkgs18.09.tar.gz $CONTAINER/root/18.09.tar.gz

# inside lxc container
tar xzfv  nixpkgs18.09.tar.gz
export NIX_PATH="nixpkgs=/root/nixpkgs-18.09:$NIX_PATH"
nix-channel --update

rowhit on 4 Jan 2019

Just another note:

[root@nixos:~]# nixos-rebuild switch
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...
mount: /dev: cannot remount devtmpfs read-write, is write-protected.
mount: /dev/pts: cannot remount devpts read-write, is write-protected.
mount: /dev/shm: cannot remount tmpfs read-write, is write-protected.
mount: /proc: cannot remount proc read-write, is write-protected.
mount: /run: cannot remount tmpfs read-write, is write-protected.
mount: /run/keys: cannot mount ramfs read-only.
mount: /run/wrappers: cannot remount tmpfs read-write, is write-protected.
Activation script snippet 'specialfs' failed (32)
setting up tmpfiles
the following new units were started: [email protected]
warning: the following units failed: sys-kernel-config.mount

● sys-kernel-config.mount - Kernel Configuration File System
   Loaded: loaded (/nix/store/kinpd6wss80pf2d04jnvvmw25lzkrlbq-systemd-239/example/systemd/system/sys-kernel-config.mount; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-01-06 12:19:58 UTC; 56ms ago
    Where: /sys/kernel/config
     What: configfs
     Docs: https://www.kernel.org/doc/Documentation/filesystems/configfs/configfs.txt
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

Jan 06 12:19:58 nixos systemd[1]: Mounting Kernel Configuration File System...
Jan 06 12:19:58 nixos mount[853]: mount: /sys/kernel/config: permission denied.
Jan 06 12:19:58 nixos systemd[1]: sys-kernel-config.mount: Mount process exited, code=exited status=32
Jan 06 12:19:58 nixos systemd[1]: sys-kernel-config.mount: Failed with result 'exit-code'.
Jan 06 12:19:58 nixos systemd[1]: Failed to mount Kernel Configuration File System.
warning: error(s) occurred while switching to the new configuration

I compared that with images:ubuntu/18.04:

root@ubuntu1:~# systemctl status sys-kernel-config.mount
● sys-kernel-config.mount - Kernel Configuration File System
   Loaded: loaded (/lib/systemd/system/sys-kernel-config.mount; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2019-01-06 11:59:57 UTC; 20min ago
    Where: /sys/kernel/config
     What: configfs
     Docs: https://www.kernel.org/doc/Documentation/filesystems/configfs/configfs.txt
           https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 52 ExecMount=/bin/mount configfs /sys/kernel/config -t configfs (code=exited, status=32)

Jan 06 11:59:57 ubuntu1 mount[52]: mount: /sys/kernel/config: permission denied.
Jan 06 11:59:57 ubuntu1 systemd[1]: sys-kernel-config.mount: Mount process exited, code=exited status=32
Jan 06 11:59:57 ubuntu1 systemd[1]: sys-kernel-config.mount: Failed with result 'exit-code'.
Jan 06 11:59:57 ubuntu1 systemd[1]: Failed to mount Kernel Configuration File System.
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

So I guess that's not critical?!

asbachb on 6 Jan 2019

Just another note. When disabling sandboxing in /etc/nix/nix.conf there's no need to mess around with proc:

sandbox = false

asbachb on 18 Jan 2019

My experience to set up NixOS in LXC container on Ubuntu without having access to the host system:

{
  config = {
    # it is not perfect because `boot.isContainer` means NixOS containers
    # not LXC, but let have something to start with
    boot.isContainer = true;

    # `boot.isContainer` implies NIX_REMOTE = "daemon" 
    # (with the comment "Use the host's nix-daemon")
    # Our host is Ubuntu, so we do not expect any "host's nix-daemon"
    environment.variables.NIX_REMOTE = lib.mkForce "";

    # (optional) suppress daemons which will vomit to the log about their unhappiness
    systemd.services."console-getty".enable = false;
    systemd.services."getty@"       .enable = false;

    # the key point (need https://github.com/NixOS/nixpkgs/issues/62856 be fixed)
    system.build.installBootLoader = pkgs.writeScript "installBootLoader.sh" ''
      #!${pkgs.bash}/bin/bash

      export TOPLEVEL="$1"
      echo "=== installBootLoader TOPLEVEL=$TOPLEVEL"

      ${pkgs.coreutils}/bin/mkdir -p /sbin
      ${pkgs.coreutils}/bin/rm -rf /sbin/init || true     # there could be symlink to "/lib/systemd/systemd"

      ${pkgs.coreutils}/bin/cat > /sbin/init <<EOF
      #!${pkgs.bash}/bin/bash

      # lustrate old OS here (otherwise ruins of /etc would prevent NixOS to boot properly)
      if [ -e "/etc/debian_version" -o -e "/etc/redhat-release" -o -e "/etc/arch-release" -o -e "/etc/gentoo-release" ]; then
        ${pkgs.coreutils}/bin/rm -rf /bin            || true
        ${pkgs.coreutils}/bin/rm -rf /etc            || true
        ${pkgs.coreutils}/bin/rm -rf /lib            || true
        ${pkgs.coreutils}/bin/rm -rf /lib64          || true
        ${pkgs.coreutils}/bin/rm -rf /snap           || true
        ${pkgs.coreutils}/bin/rm -rf /usr            || true
        ${pkgs.coreutils}/bin/rm -rf /var            || true
      fi

      exec $TOPLEVEL/init
      EOF

      ${pkgs.coreutils}/bin/chmod 0755 /sbin/init
    '';
  };
}

Besides https://github.com/NixOS/nixpkgs/issues/62856 I used 2 more patches of <nixpkgs>.
The first is required to run Tinc VPN inside LXC container, the second fixes "nixos-rebuild switch".
Both are optional for the first install

# LXC: two replaces (in LXC container /dev/net/tun is pre-available, "dev-net-tun.device" always fails)
substituteInPlace nixos/modules/tasks/network-interfaces-scripted.nix \
  --replace '[ "dev-net-tun.device" ' \
            'optionals (!config.boot.isContainer) [ "dev-net-tun.device" ] ++ [ '

# LXC: fix "Failed to mount Kernel Configuration File System." on "nixos-rebuild switch"
substituteInPlace nixos/modules/system/boot/systemd.nix \
  --replace '"sys-kernel-config.mount"'         '] ++ (optional (!config.boot.isContainer) "sys-kernel-config.mount"      ) ++ [' \
  --replace '"systemd-journald-audit.socket"'   '] ++ (optional (!config.boot.isContainer) "systemd-journald-audit.socket") ++ ['

volth on 8 Jun 2019

👍3

@volth cold you post a log of your nixos-rebuild switch? I applied your patches but seems that nixos-rebuild switch still does not work for me:

[root@nixos:~]# nixos-rebuild switch
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/ls4sspqmwsbpg466y5v8qg47vfcq64kp-system-units.drv
  /nix/store/2y806mgkbd4lc5q5zy28bk4n6ps7yqpp-etc.drv
  /nix/store/0wmh6v4fbkh6549scnlq1fr748fpidr1-nixos-system-nixos-19.03.173481.8a30e242181.drv
building '/nix/store/ls4sspqmwsbpg466y5v8qg47vfcq64kp-system-units.drv'...
building '/nix/store/2y806mgkbd4lc5q5zy28bk4n6ps7yqpp-etc.drv'...
building '/nix/store/0wmh6v4fbkh6549scnlq1fr748fpidr1-nixos-system-nixos-19.03.173481.8a30e242181.drv'...
stopping the following units: sys-kernel-debug.mount
activating the configuration...
setting up /etc...
mount: /dev: cannot remount devtmpfs read-write, is write-protected.
mount: /dev/pts: cannot remount devpts read-write, is write-protected.
mount: /dev/shm: cannot remount tmpfs read-write, is write-protected.
mount: /proc: cannot remount proc read-write, is write-protected.
mount: /run: cannot remount tmpfs read-write, is write-protected.
mount: /run/keys: cannot remount ramfs read-write, is write-protected.
mount: /run/wrappers: cannot remount tmpfs read-write, is write-protected.
Activation script snippet 'specialfs' failed (32)
reloading user units for root...
setting up tmpfiles
warning: error(s) occurred while switching to the new configuration

asbachb on 12 Sep 2019

I get this error when I try to start a container as described in this thread:

lxc nixos-test 20191125172124.800 WARN     initutils - initutils.c:setproctitle:341 - Invalid argument - Failed to set cmdline
lxc nixos-test 20191125172124.954 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 1
lxc nixos-test 20191125172124.954 ERROR    start - start.c:lxc_init:887 - Failed to run lxc.hook.pre-start for container "nixos-test"
lxc nixos-test 20191125172124.954 ERROR    start - start.c:__lxc_start:1988 - Failed to initialize container "nixos-test"
lxc nixos-test 20191125172124.980 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:864 - No such file or directory - Failed to receive the container state

metadata.yaml:

architecture: "x86_64"
creation_date: 1574699218
properties:
  architecture: "x86_64"
  description: "NixOS 19.09"
  os: "nixos"
  release: "19.09"
templates:
  /etc/hostname:
    when:
      - create
      - copy
    template: hostname.tpl

templates/hostname.tpl:

{{ container.name }}

lxc profile show default-nixos:

lxc profile show default-nixos
config:
  raw.lxc: |-
    lxc.init.cmd = /init
    lxc.apparmor.allow_incomplete = 1
description: Default profile for NixOS
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: lxd
    type: disk
name: default-nixos
used_by:
- /1.0/containers/nixos-test

Any suggestions? At least, how can I get more debugging information out of lxd?

Update:

I just caught this in the system log:

lxd[983]: t=2019-11-25T11:56:32-0600 lvl=eror msg="The start hook failed" container=nixos-test err="open /var/lib/lxd/containers/nixos-test/rootfs/etc/hostname: no such file or directory"

This seems to support the hypothesis that there's a problem with the metadata.yaml or templates that I gave, but I don't know what.

Update:

I made some progress by removing the hostname template; now NixOS fails to boot with this output:

[1;32m<<< NixOS Stage 2 >>>[0m

mount: /nix/store: cannot mount /dev/disk/by-uuid/3c1ec4f3-e3c5-42dd-ac3a-b247322cb0a0 read-only.
touch: cannot touch '/etc/fstab': Too many levels of symbolic links
running activation script...
setting up /etc...
/etc/static directory contains user files. Symlinking may fail. at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 99.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 110.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 113.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 114.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 115.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 110.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 113.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 114.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 115.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 110.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 113.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 114.
Warning: something's wrong at /nix/store/vn1xs9s2akf79y1pbya2qldydbf2b22m-setup-etc.pl line 115.
mount: /dev: cannot remount devtmpfs read-write, is write-protected.
mount: /dev/pts: cannot remount devpts read-write, is write-protected.
mount: /proc: cannot remount proc read-write, is write-protected.
mount: /run/keys: WARNING: device write-protected, mounted read-only.
Activation script snippet 'specialfs' failed (32)
starting systemd...
Failed to open configuration file '/etc/systemd/system.conf': Too many levels of symbolic links
systemd 243 running in system mode. (+PAM +AUDIT -SELINUX +IMA +APPARMOR +SMACK -SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.
Failed to read os-release file, ignoring: Too many levels of symbolic links

Welcome to Linux!

Failed to read configured hostname: Too many levels of symbolic links
Failed to read /etc/locale.conf: Too many levels of symbolic links
Failed to read /etc/locale.conf: Too many levels of symbolic links
Failed to open generator directory /etc/systemd/system-generators: Too many levels of symbolic links
/nix/store/f1lpfdlzqa5l711zmkg32zm6nrxp95m6-systemd-243/lib/systemd/system-generators/systemd-fstab-generator failed with exit status 1.
Failed to open "/etc/systemd/system", ignoring: Too many levels of symbolic links
Unit default.target not found.
Falling back to rescue target: rescue.target
Unit rescue.target not found.
[[0;1;31m!!!!!![0m] Failed to load rescue target.
Exiting PID 1...

I notice lots of Too many levels of symbolic links but I don't know what that could be about.

ttuegel on 25 Nov 2019

With #74378 I have run switch-to-configuration successfully in a NixOS container on an Ubuntu host, using this LXC profile:

config:
  raw.lxc: |-
    lxc.init.cmd = /init
    lxc.apparmor.allow_incomplete = 1
    lxc.apparmor.profile = unconfined
description: Default profile for NixOS
devices:
  eth0:
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: lxd
    type: disk

The setting lxc.apparmor.profile = unconfined is essential for remounting /nix/store read-only. For some reason, LXC considers read-only remounting a bind mount to be privileged, even if the mount only exists inside the container. If there are security concerns, we could also make a custom profile which allows read-only remounting bind mounts.

ttuegel on 27 Nov 2019

Thank you for your contributions.
This has been automatically marked as stale because it has had no activity for 180 days.
If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.
Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the
related code and @ mention them in a comment.
Ask on the NixOS Discourse. 3. Ask on the #nixos channel on
irc.freenode.net.