Nixpkgs: lxd: fails to launch containers

[mt-caret]

Issue description

Trying to launch a container produces an error.

$ lxc info --show-log test | grep ERROR

lxc 20170428072645.996 ERROR    lxc_conf - conf.c:instantiate_veth:2669 - failed to attach 'veth3H0S6D' to the bridge 'lxdbr0': Operation not permitted
lxc 20170428072646.944 ERROR    lxc_conf - conf.c:lxc_create_network:2962 - failed to create netdev
lxc 20170428072646.945 ERROR    lxc_start - start.c:lxc_spawn:1088 - Failed to create the network.
lxc 20170428072646.946 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "test"

Steps to reproduce

set virtualisation.lxd.enable = true; and add use to group "lxd", then run $ lxc launch ubuntu:16.04 test

Technical details

• System: (NixOS: nixos-version, Ubuntu/Fedora: lsb_release -a, ...)
  17.09pre106117.c90998d5cf (Hummingbird)
• Nix version: (run nix-env --version)
  nix-env (Nix) 1.11.8
• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion)
  "17.09pre106117.c90998d5cf"
• lxd version
  2.0.2

[mt-caret]

Solved this a while ago with:

1. sudo mkdir -p /var/lib/lxc/rootfs
2. enable apparmor in configuration.nix (https://github.com/raphael/linux-samus/issues/103)

I'm not 100% sure if both of these are required or are still relevant, so please keep that in mind.

EDIT: These instructions are misleading/inaccurate. Please refer to https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-364625022.

[fpletz]

We should fix this in the lxd module.

[mt-caret]

I would love to, but I don't have much free time right now; salvaging what looks relevant from my configuration.nix gives me

security.apparmor = {
  enable = true;
  profiles = [
    "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start" 
    "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
  ];
  packages = [ pkgs.lxc ];
}

EDIT: fixed typo in code lxc-star -> lxc-start

[stefano-m]

Just a note to say that although I have added the configuration suggested in https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-316971140 I still cannot start containers.

Specifically, lxc tells me that my kernel does not support Apparmor

System information:

nixos-version
17.09.2182.7f6f0c49f0 (Hummingbird)

nix-env --version
nix-env (Nix) 1.11.15

nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion
"17.09.2182.7f6f0c49f0"

grep linuxPackages /etc/nixos/hardware-configuration.nix
  boot.kernelPackages = pkgs.linuxPackages_hardened;

uname -r
4.13.16

(but I have also tried unsuccessfully with the stock kernel and the 4.13 one)

# /etc/nixos/lxd.conf
{ config, pkgs, ... }:

{ 
  virtualisation.lxd.enable = true;
  virtualisation.lxc.enable = true;

  # https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-316971140
  security.apparmor = {
    enable = true;
    profiles = [
      "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start"
      "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
    ];
    packages = [ pkgs.lxc ];
  };
}

(I also tried to add virtualisation.lxc.defaultConfig = "lxc.aa_allow_incomplete = 1"; to no avail)

lxc --version
2.16

lxd --version
2.16

Error log

lxc launch images:alpine/3.7/amd64 testcontainer1

lxc info --show-log local:testcontainer1
Name: testcontainer1
Remote: unix://
Architecture: x86_64
Created: 2017/12/15 16:17 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

            lxc 20171215161722.864 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:218 - Incomplete AppArmor support in your kernel
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:220 - If you really want to start this container, set
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:221 - lxc.aa_allow_incomplete = 1
            lxc 20171215161722.864 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:222 - in your container configuration file
            lxc 20171215161722.864 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
            lxc 20171215161722.864 ERROR    lxc_start - start.c:__lxc_start:1354 - Failed to spawn container "testcontainer1".
            lxc 20171215161722.881 ERROR    lxc_conf - conf.c:run_buffer:408 - Script exited with status 1.
            lxc 20171215161722.881 ERROR    lxc_start - start.c:lxc_fini:546 - Failed to run lxc.hook.post-stop for container "testcontainer1".
            lxc 20171215161722.881 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:177 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20171215161722.882 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1253 - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd//lxc/testcontainer1: Device or resource busy
            lxc 20171215161722.882 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/systemd//lxc/testcontainer1
            lxc 20171215161722.882 WARN     lxc_cgfsng - cgroups/cgfsng.c:cgroup_rmdir:1253 - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory//lxc/testcontainer1: Device or resource busy
            lxc 20171215161722.882 ERROR    lxc_cgfsng - cgroups/cgfsng.c:recursive_destroy:1288 - Error destroying /sys/fs/cgroup/memory//lxc/testcontainer1

[mt-caret]

I reinstalled NixOS on my laptop for an unrelated reason so I was forced to go through setting up lxd again; I'm documenting what works along with links that helped. Hopefully someone with a bit more NixOS expertise can update the module so we don't have to deal with this.

Apologies to @stefano-m; https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-316172360 was kind of misleading.

1. create missing directory: sudo mkdir -p /var/lib/lxc/rootfs

This will give you errors as shown here: https://github.com/NixOS/nixpkgs/issues/25280#issuecomment-352047373 and https://github.com/raphael/linux-samus/issues/103

1. allow incomplete apparmor lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 as shown here: https://github.com/raphael/linux-samus/issues/103#issuecomment-210005060

This will now give you these messages as shown here: https://github.com/lxc/lxc/issues/1895

lxc 20180210034825.988 WARN     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:218 - Incomplete AppArmor support in your kernel
lxc 20180210034825.988 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:234 - No such file or directory - failed to change apparmor profile to lxc-container-default-cgns

The lxc issue in question has a comment with a workaround that seems to involve disabling apparmor altogether, but this can also be fixed by adding the relevant apparmor profiles (I can't remember where or how I found this information).

1. add profiles for lxc in configuration.nix

security.apparmor = {
  enable = true;
  profiles = [
    "${pkgs.lxc}/etc/apparmor.d/usr.bin.lxc-start"
    "${pkgs.lxc}/etc/apparmor.d/lxc-containers"
  ];
  packages = [ pkgs.lxc ];
};

Now lxc launch ubuntu:16.04 should work without any problems.

[mt-caret]

Now https://github.com/NixOS/nixpkgs/pull/34803 has been merged, I believe all you need to do to get lxd up and running is to enable lxd in configuration.nix, lxd init, and lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 AFAIK (would love if someone can confirm this).

[mt-caret]

This issue seems relevant: https://github.com/lxc/lxd/issues/2834

[mt-caret]

As of 18.09, lxc profile set default raw.lxc lxc.aa_allow_incomplete=1 no longer seems to be necessary.

[mt-caret]

AFAIK this is now solved (virtualisation.lxd.enable = true will now work without additional effort) so closing.