Lxd: Container rootfs mounting fails since the wrong storage engine is used (dir instead of btrfs)

Created on 26 Dec 2017  路  10Comments  路  Source: lxc/lxd

Required information

  • Distribution: Ubuntu Server
  • Distribution version: 16.04.3
  • The output of "lxc info" or if that fails:
config:
  core.https_address: 127.0.0.1:65252
  core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 127.0.0.1:65252
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIFWDCCA0CgAwIBAgIQJiR1TzIiszDq9HQHoYRF3jANBgkqhkiG9w0BAQsFADA8
    MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRwwGgYDVQQDDBNyb290QGRl
    ZGktcGFyLTcwODA0MB4XDTE3MTEzMDIxMzkzOFoXDTI3MTEyODIxMzkzOFowPDEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEcMBoGA1UEAwwTcm9vdEBkZWRp
    LXBhci03MDgwNDCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANl1/MS4
    5VjW4aQyCz3auK+F7aVJk3dYrnV7OSNPrPhBlpc6bt/eY9lNcKNxXkDAyQLX1ZHe
    TvkJyweUdVbXxVgodKKvmltdJ0Z/cSeaCSeQqn+1vbHWaOk2ZYHzXBndEFBsVKHD
    HZCFiowEs0/3xEVZZvmREgkSRZN8rHxjMFYW78mRH2KDrDhDiLInmUXqNKe7sHxR
    7XJs+Z4HyLclaZ1xXcshNtAh4yR2DyU3zWxRdfj9fmkX2i/Y+fjfokvb/SjcgiQj
    LNfUdaYQ607BQk1gOzBy2LqOpF+fjDpEpQuHeiHxeLAg6b9w6ullA5FoyNOxQRbS
    U8W3ZeejouWIM8xvnHXLXxK7v2pwJUP+Fb5N0Rihl0ugKWRdQoQry8LI/vZUWtXw
    pAYpOBuqjJ9LQUhJjWNWwuulOSgYfijWwRHzs07itnOZ2p7Ur/j1D2zi1k4awBp6
    l2qWrBEV2BsWP2UA30p79dZkMozpSBcxnJBDYKc9ZCE/lx14hKm3jRVCAYl6ax11
    kZo6FpXlNKaIIUaFwXP0zww5m7ymbdFqkrELOyMy6f6qpAzo8SqCh9JykSWl0T5N
    eFJyKHqixDVY7/ho6yf48GfXvVXds96l7cYCpK8FqRBGXO0Zzv6Tp6Lf5tjbFran
    alzFlxLNHub0TWDbcy6Nzqnu95pcvaZbftpRAgMBAAGjVjBUMA4GA1UdDwEB/wQE
    AwIFoDATBgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB8GA1UdEQQY
    MBaCDmRlZGktcGFyLTcwODA0hwQ+0kUJMA0GCSqGSIb3DQEBCwUAA4ICAQC/eOpA
    fymnVhaqWMVswfxXcg3QEb2NEEQrTtayFs9/PhoYMhsxwNiqkt+dyG6sHhzQ7kgS
    mTGoUsi+xZwds9i40uGwR8QKEw85vIdVHrGfreJTWCmN3K0SVEW38Ot/WbT0xfGJ
    tV6wpdXKQs0lRdIbXKZaS6zTIa1Bb2XtOmrKSf0QFjgcNipdWHvfOEF/vCzqW3YJ
    3pSlgrnD5GZlzUKFAzhcJVCSkSOSpbnnOMNRCcAbVTz0ABlUSyoObuNa5rr3Lht8
    DYzuMgTcYoC+FLdpSUTGuF3KOit1ojqKDOREQ6bymoRt73HOEyEsIjg+E8QWZis6
    37X8s54nRDS46vDcxgptm0DSPgIf0YaxSuKsg04itFehNqfgXQdUfR3xcQ+hROUR
    BCuehQSD4RdDmLq0s0eUHtdg3MCMEKeuUTPYyYkSzIe+ykF+/kpOfB8TFaVJheGs
    7sjvy7QksCRPN5lXzrBDYPozcF9Nsyd0JzEv3+FGrbqKAabSh0HFlwMojKGMnjdj
    V8S3/bW8f4ELcsBUKbTk2kTeoVqo2TuUtJ1on+yZanyQcgYW5EegVB213rzDotW+
    nAhIi2oAxXJSCYeyYOv752Zt3L3qOmySc2RWnL1hlCZaQKmaWiynt4ugj8L+6hpO
    UkZS11YD4p2972hb2vdL8a21vHTuT1qxXT4kYA==
    -----END CERTIFICATE-----
  certificate_fingerprint: e93a0562395b2052d599602a1820c28ced2cae4390038bbae3233a9f3dcbb15d
  driver: lxc
  driver_version: 2.1.1
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-103-generic
  server: lxd
  server_pid: 979
  server_version: "2.21"
  storage: btrfs
  storage_version: "4.4"

Issue description

Mounting a containers rootfs fails (creating new container, starting existing) since the wrong storage engine is selected.

Steps to reproduce (might be)

  1. Mount /var/lib/lxd as an btrfs volume
  2. Create a container

Information to attach

  • [x] Any relevant kernel output (dmesg)
[56994.048156] audit: type=1400 audit(1514323470.080:51): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-ubuntu-minimal_</var/lib/lxd>" pid=21426 comm="apparmor_parser"
[56994.076374] device vethF4R9UU entered promiscuous mode
[56994.076501] IPv6: ADDRCONF(NETDEV_UP): vethF4R9UU: link is not ready
[56994.141081] lxdbr0: port 2(vethF4R9UU) entered disabled state
[56994.143474] device vethF4R9UU left promiscuous mode
[56994.143480] lxdbr0: port 2(vethF4R9UU) entered disabled state
[56995.185820] audit: type=1400 audit(1514323471.216:52): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-ubuntu-minimal_</var/lib/lxd>" pid=21556 comm="apparmor_parser"
  • [x] Container log (lxc info NAME --show-log)
lxc info --show-log ubuntu-minimal
Name: ubuntu-minimal
Remote: unix://
Architecture: x86_64
Created: 2017/12/21 11:35 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

            lxc 20171226112643.325 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112643.325 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112643.325 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112643.325 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112643.325 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112643.325 TRACE    lxc_start - start.c:lxc_init_handler:583 - Created anonymous pair {4,6} of unix sockets
            lxc 20171226112643.325 TRACE    lxc_start - start.c:lxc_init_handler:592 - unix domain socket 8 for command server is ready
            lxc 20171226112643.326 INFO     lxc_container - lxccontainer.c:do_lxcapi_start:883 - Attempting to set proc title to [lxc monitor] /var/lib/lxd/containers ubuntu-minimal
            lxc 20171226112643.327 INFO     lxc_start - start.c:lxc_check_inherited:257 - Closed inherited fd: 3.
            lxc 20171226112643.327 INFO     lxc_start - start.c:lxc_check_inherited:257 - Closed inherited fd: 7.
            lxc 20171226112643.327 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
            lxc 20171226112643.327 TRACE    lxc_start - start.c:lxc_init:607 - initialized LSM
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for reject_force_umount action 0(kill).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for reject_force_umount action 0(kill).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .[all].
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .kexec_load errno 38.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for kexec_load action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for kexec_load action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .open_by_handle_at errno 38.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for open_by_handle_at action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for open_by_handle_at action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .init_module errno 38.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for init_module action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for init_module action 327718(errno).
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .finit_module errno 38.
            lxc 20171226112643.327 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for finit_module action 327718(errno).
            lxc 20171226112643.328 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for finit_module action 327718(errno).
            lxc 20171226112643.328 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .delete_module errno 38.
            lxc 20171226112643.328 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for delete_module action 327718(errno).
            lxc 20171226112643.328 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for delete_module action 327718(errno).
            lxc 20171226112643.328 INFO     lxc_seccomp - seccomp.c:parse_config_v2:624 - Merging in the compat Seccomp ctx into the main one.
            lxc 20171226112643.328 TRACE    lxc_start - start.c:lxc_init:613 - read seccomp policy
            lxc 20171226112643.328 TRACE    lxc_start - start.c:lxc_serve_state_clients:373 - set container state to STARTING
            lxc 20171226112643.328 TRACE    lxc_start - start.c:lxc_serve_state_clients:376 - no state clients registered
            lxc 20171226112643.328 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.328 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.328 TRACE    lxc_start - start.c:lxc_init:620 - set container state to "STARTING"
            lxc 20171226112643.328 TRACE    lxc_start - start.c:lxc_init:648 - set environment variables
            lxc 20171226112643.328 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/bin/lxd callhook /var/lib/lxd 18 start" for container "ubuntu-minimal", config section "lxc".
            lxc 20171226112643.398 TRACE    lxc_start - start.c:lxc_init:654 - ran pre-start hooks
            lxc 20171226112643.398 DEBUG    lxc_start - start.c:setup_signal_fd:301 - Set SIGCHLD handler with file descriptor: 3.
            lxc 20171226112643.398 TRACE    lxc_start - start.c:lxc_init:665 - set up signal fd
            lxc 20171226112643.399 DEBUG    console - console.c:lxc_console_peer_default:450 - process does not have a controlling terminal
            lxc 20171226112643.399 TRACE    lxc_start - start.c:lxc_init:672 - created console
            lxc 20171226112643.399 TRACE    lxc_conf - conf.c:lxc_ttys_shift_ids:2969 - chowned console "/dev/pts/3"
            lxc 20171226112643.399 TRACE    lxc_start - start.c:lxc_init:678 - shifted tty ids
            lxc 20171226112643.399 INFO     lxc_start - start.c:lxc_init:680 - container "ubuntu-minimal" is initialized
            lxc 20171226112643.399 DEBUG    storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
            lxc 20171226112643.402 INFO     lxc_network - network.c:instantiate_veth:171 - Retrieved mtu 1500 from lxdbr0
            lxc 20171226112643.403 INFO     lxc_network - network.c:instantiate_veth:197 - Attached "vethM1LLU7" to bridge "lxdbr0"
            lxc 20171226112643.403 DEBUG    lxc_network - network.c:instantiate_veth:214 - Instantiated veth "vethM1LLU7/veth7WNWCG", index is "13"
            lxc 20171226112643.403 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:67 - cgroup driver cgroupfs-ng initing for ubuntu-minimal
            lxc 20171226112643.403 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112643.403 TRACE    lxc_commands - commands.c:lxc_cmd_send:222 - Command "get_cgroup" connected to command socket
            lxc 20171226112643.403 TRACE    lxc_commands - commands.c:lxc_cmd_send:238 - Command "get_cgroup" requested data of length 8
            lxc 20171226112643.403 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:468 - No isolated cpus detected.
            lxc 20171226112643.403 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:639 - "cgroup.clone_children" was already set to "1".
            lxc 20171226112643.404 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWUSER.
            lxc 20171226112643.404 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWNS.
            lxc 20171226112643.404 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWPID.
            lxc 20171226112643.404 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWUTS.
            lxc 20171226112643.404 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWIPC.
            lxc 20171226112643.404 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.404 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.404 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.409 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16921 0 165536 65536"
            lxc 20171226112643.413 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16921 0 165536 65536"
            lxc 20171226112643.414 INFO     lxc_start - start.c:do_start:848 - Unshared CLONE_NEWNET.
            lxc 20171226112643.434 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16961" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.434 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16961" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.434 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16961" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.434 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16961" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.434 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.434 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.434 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.438 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16961 0 165536 65536 65536 0 1"
            lxc 20171226112643.442 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16961 0 165536 65536 65536 0 1"
            lxc 20171226112643.442 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "chown_cgroup_wrapper"
            lxc 20171226112643.453 DEBUG    lxc_network - network.c:lxc_network_move_created_netdev_priv:2445 - Moved network device "veth7WNWCG"/"eth0" to network namespace of 16921
            lxc 20171226112643.453 NOTICE   lxc_utils - utils.c:lxc_switch_uid_gid:2073 - Switched to gid 0.
            lxc 20171226112643.453 NOTICE   lxc_utils - utils.c:lxc_switch_uid_gid:2079 - Switched to uid 0.
            lxc 20171226112643.453 NOTICE   lxc_utils - utils.c:lxc_setgroups:2091 - Dropped additional groups.
            lxc 20171226112643.453 INFO     lxc_start - start.c:do_start:925 - Unshared CLONE_NEWCGROUP.
            lxc 20171226112643.454 DEBUG    storage - storage/storage.c:get_storage_by_name:231 - Detected rootfs type "dir"
            lxc 20171226112643.454 ERROR    dir - storage/dir.c:dir_mount:179 - Permission denied - Failed to mount "/var/lib/lxd/containers/ubuntu-minimal/rootfs" on "/usr/lib/x86_64-linux-gnu/lxc"
            lxc 20171226112643.454 ERROR    lxc_conf - conf.c:lxc_setup_rootfs:1313 - Failed to mount rootfs "dir:/var/lib/lxd/containers/ubuntu-minimal/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc" with options "(null)".
            lxc 20171226112643.454 ERROR    lxc_conf - conf.c:do_rootfs_setup:3105 - failed to setup rootfs for 'ubuntu-minimal'
            lxc 20171226112643.454 ERROR    lxc_conf - conf.c:lxc_setup:3146 - Error setting up rootfs mount after spawn
            lxc 20171226112643.454 ERROR    lxc_start - start.c:do_start:944 - Failed to setup container "ubuntu-minimal".
            lxc 20171226112643.454 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
            lxc 20171226112643.454 INFO     lxc_network - network.c:lxc_delete_network_priv:2529 - Interface "eth0" with index 13 already deleted or existing in different network namespace
            lxc 20171226112643.454 INFO     lxc_network - network.c:lxc_delete_network_priv:2539 - Removed interface "eth0" with index 13
            lxc 20171226112643.493 INFO     lxc_network - network.c:lxc_delete_network_priv:2560 - Removed interface "vethM1LLU7" from "lxdbr0"
            lxc 20171226112643.493 DEBUG    lxc_network - network.c:lxc_delete_network:3124 - Deleted network devices
            lxc 20171226112643.493 TRACE    lxc_start - start.c:lxc_serve_state_socket_pair:446 - Sent container state "ABORTING" to 6
            lxc 20171226112643.493 TRACE    lxc_start - start.c:lxc_serve_state_clients:373 - set container state to ABORTING
            lxc 20171226112643.493 TRACE    lxc_start - start.c:lxc_serve_state_clients:376 - no state clients registered
            lxc 20171226112643.493 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.493 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:760 - Received container state "ABORTING" instead of "RUNNING"
            lxc 20171226112643.493 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.493 ERROR    lxc_start - start.c:__lxc_start:1459 - Failed to spawn container "ubuntu-minimal".
            lxc 20171226112643.493 TRACE    lxc_start - start.c:lxc_serve_state_clients:373 - set container state to STOPPING
            lxc 20171226112643.493 TRACE    lxc_start - start.c:lxc_serve_state_clients:376 - no state clients registered
            lxc 20171226112643.493 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.493 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.517 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16975" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.517 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16975" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.517 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16975" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.517 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16975" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.517 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.517 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.517 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.521 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16975 0 165536 65536 65536 0 1"
            lxc 20171226112643.525 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16975 0 165536 65536 65536 0 1"
            lxc 20171226112643.525 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.526 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16980" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.526 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16980" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.526 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16980" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.526 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16980" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.526 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.526 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.526 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.530 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16980 0 165536 65536 65536 0 1"
            lxc 20171226112643.534 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16980 0 165536 65536 65536 0 1"
            lxc 20171226112643.535 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.536 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16985" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.536 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16985" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.536 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16985" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.536 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16985" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.536 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.536 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.536 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.540 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16985 0 165536 65536 65536 0 1"
            lxc 20171226112643.543 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16985 0 165536 65536 65536 0 1"
            lxc 20171226112643.544 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.545 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16990" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.545 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16990" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.545 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16990" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.545 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16990" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.545 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.545 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.545 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.549 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16990 0 165536 65536 65536 0 1"
            lxc 20171226112643.553 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16990 0 165536 65536 65536 0 1"
            lxc 20171226112643.553 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.554 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16995" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.554 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "16995" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.554 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16995" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.554 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "16995" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.554 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.554 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.554 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.558 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 16995 0 165536 65536 65536 0 1"
            lxc 20171226112643.562 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 16995 0 165536 65536 65536 0 1"
            lxc 20171226112643.562 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.563 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17000" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.563 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17000" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.563 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17000" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.563 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17000" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.563 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.563 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.563 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.567 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17000 0 165536 65536 65536 0 1"
            lxc 20171226112643.571 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17000 0 165536 65536 65536 0 1"
            lxc 20171226112643.571 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.572 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17005" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.572 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17005" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.572 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17005" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.572 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17005" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.572 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.572 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.572 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.576 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17005 0 165536 65536 65536 0 1"
            lxc 20171226112643.580 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17005 0 165536 65536 65536 0 1"
            lxc 20171226112643.580 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.581 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17010" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.581 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17010" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.581 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17010" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.581 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17010" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.581 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.581 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.581 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.585 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17010 0 165536 65536 65536 0 1"
            lxc 20171226112643.590 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17010 0 165536 65536 65536 0 1"
            lxc 20171226112643.590 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.591 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17015" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.591 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17015" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.591 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17015" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.591 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17015" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.591 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.591 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.591 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.595 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17015 0 165536 65536 65536 0 1"
            lxc 20171226112643.599 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17015 0 165536 65536 65536 0 1"
            lxc 20171226112643.599 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.600 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17021" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.600 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17021" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.600 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17021" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.600 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17021" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.600 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.600 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.600 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.604 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17021 0 165536 65536 65536 0 1"
            lxc 20171226112643.609 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17021 0 165536 65536 65536 0 1"
            lxc 20171226112643.609 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.610 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17026" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.610 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing uid mapping for "17026" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.610 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17026" in new user namespace: nsuid 0 - hostid 165536 - range 65536
            lxc 20171226112643.610 TRACE    lxc_conf - conf.c:userns_exec_1:3816 - establishing gid mapping for "17026" in new user namespace: nsuid 65536 - hostid 0 - range 1
            lxc 20171226112643.610 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newuidmap" does have the setuid bit set.
            lxc 20171226112643.610 DEBUG    lxc_conf - conf.c:idmaptool_on_path_and_privileged:2595 - The binary "/usr/bin/newgidmap" does have the setuid bit set.
            lxc 20171226112643.610 DEBUG    lxc_conf - conf.c:lxc_map_ids:2683 - Functional newuidmap and newgidmap binary found.
            lxc 20171226112643.614 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newuidmap wrote mapping "newuidmap 17026 0 165536 65536 65536 0 1"
            lxc 20171226112643.618 TRACE    lxc_conf - conf.c:lxc_map_ids:2739 - newgidmap wrote mapping "newgidmap 17026 0 165536 65536 65536 0 1"
            lxc 20171226112643.618 TRACE    lxc_conf - conf.c:run_userns_fn:3576 - calling function "rmdir_wrapper"
            lxc 20171226112643.618 TRACE    lxc_start - start.c:lxc_serve_state_clients:373 - set container state to STOPPED
            lxc 20171226112643.618 TRACE    lxc_start - start.c:lxc_serve_state_clients:376 - no state clients registered
            lxc 20171226112643.618 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.618 WARN     lxc_monitor - monitor.c:lxc_monitor_fifo_send:111 - Failed to open fifo to send message: No such file or directory.
            lxc 20171226112643.619 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "ubuntu-minimal", config section "lxc".
            lxc 20171226112643.619 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:125 - Command get_cgroup failed to receive response: Connection reset by peer.
            lxc 20171226112643.619 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection reset by peer.
            lxc 20171226112643.619 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112643.619 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112644.123 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/bin/lxd callhook /var/lib/lxd 18 stop" for container "ubuntu-minimal", config section "lxc".
            lxc 20171226112644.165 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112644.165 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112644.165 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112644.165 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112644.165 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112644.530 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112644.530 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112644.530 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112644.530 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112644.530 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112907.533 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112907.533 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112907.533 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112907.533 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112907.533 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112907.542 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112907.542 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112907.542 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112907.542 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_cgroup tries to connect command socket
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_cgroup failed to connect command socket: Connection refused
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd_get_cgroup_path:440 - command get_cgroup failed for container "ubuntu-minimal": Connection refused.
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd:290 - command get_state tries to connect command socket
            lxc 20171226112907.543 TRACE    lxc_commands - commands.c:lxc_cmd:295 - command get_state failed to connect command socket: Connection refused

All 10 comments

Now I'm a little bit unsure if this is connected with a wrong detecetd storage type. I've one running container. A diffrerence which I can see is that it's a privilged container:

/var/lib/lxd/storage-pools/default/containers# ls -l
total 0
drwxr-xr-x  1 root   root   34 Dec 23 16:43 ubuntu-mail
drwxr-xr-x+ 1 165536 165536 78 Dec 21 12:35 ubuntu-minimal

So

  • ubuntu-mail is starting fine
  • ubuntu-minimal does not

Note that putitng your /var/lib/lxd on btrfs will NOT make LXD use btrfs optimized container storage.

The only way it would do so is if your storage pool is using the btrfs driver (you can check that with lxc storage list).

The most likely issue here is that your permissions on /var/lib/lxd itself are wrong as a result of you manually mounting a filesystem over it.

They should at a minimum be 0711, otherwise unprivileged containers will not be able to start (as is the case for you).

lxc storage indicates that btrfs should be used:

lxc storage list
+---------+-------------+--------+------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |               SOURCE               | USED BY |
+---------+-------------+--------+------------------------------------+---------+
| default |             | btrfs  | /var/lib/lxd/storage-pools/default | 10      |
+---------+-------------+--------+------------------------------------+---------+

The /var/lib/lxd folder seems to have the minimum of the required permissions:

ls -l /var/lib | grep lxd
drwxr-xr-x 1 root root  234 Dec 26 22:24 lxd

Can you tell me the permissions on the following:

  • /var
  • /var/lib
  • /var/lib/lxd/containers
  • /var/lib/lxd/storage-pools
  • /var/lib/lxd/storage-pools/default
  • /var/lib/lxd/storage-pools/default/containers
  • /var/lib/lxd/containers//ubuntu-minimal
  • /var/lib/lxd/containers/ubtuntu-minimal/rootfs

That should tell us why you're getting that permission error.

Note that lxc.log showing the "dir" backend is a red herring, that's because LXD always does the mounting itself and so liblxc then is always told to just treat the container as a plain old directory.

I'm a little curious what happened since it worked several weeks without problems...

ls -l / | grep var
drwxr-xr-x  12 root root  4096 Dec  2 13:36 var
ls -l /var | grep lib
drwxr-xr-x 48 root root   4096 Dec 16 01:41 lib



md5-b05cd2f189a204797aa7ad765a71a495



ls -ls /var/lib/lxd/
total 76
 0 drwx--x--x 1 root root   110 Dec 26 05:05 containers
 0 drwx--x--x 1 root root   110 Dec 26 05:05 devices
 0 drwxr-xr-x 2 root root    60 Dec 26 12:26 devlxd
 0 drwx------ 1 root root     0 Nov 30 22:39 disks
 0 drwx------ 1 root root   796 Dec 26 06:25 images
68 -rw-r--r-- 1 root root 66560 Dec 26 22:24 lxd.db
 0 drwx--x--x 1 root root    12 Dec  7 20:01 networks
 0 drwx------ 1 root root    30 Dec  3 09:58 security
 4 -rw-r--r-- 1 root root  1915 Nov 30 22:39 server.crt
 4 -rw------- 1 root root  3243 Nov 30 22:39 server.key
 0 drwx--x--x 4 root root    80 Dec 26 06:36 shmounts
 0 drwx------ 1 root root     0 Nov 30 22:39 snapshots
 0 drwx--x--x 1 root root    14 Dec 26 06:52 storage-pools
 0 srw-rw---- 1 root lxd      0 Dec 26 06:34 unix.socket



md5-b05cd2f189a204797aa7ad765a71a495



ls -ls /var/lib/lxd/storage-pools/
total 0
0 drwxr-xr-x 1 root root 62 Nov 30 22:43 default



md5-b05cd2f189a204797aa7ad765a71a495



ls -ls /var/lib/lxd/storage-pools/default/
total 0
0 drwx------ 1 root root 110 Dec 26 05:05 containers
0 drwxr-xr-x 1 root root   0 Nov 30 22:43 custom
0 drwxr-xr-x 1 root root 512 Dec 26 06:25 images
0 drwxr-xr-x 1 root root   0 Nov 30 22:43 snapshots



md5-b05cd2f189a204797aa7ad765a71a495



ls -ls /var/lib/lxd/containers/
total 16
4 lrwxrwxrwx 1 root root 57 Dec 23 16:41 ubuntu-mail -> /var/lib/lxd/storage-pools/default/containers/ubuntu-mail
4 lrwxrwxrwx 1 root root 60 Dec 21 12:35 ubuntu-minimal -> /var/lib/lxd/storage-pools/default/containers/ubuntu-minimal



md5-b05cd2f189a204797aa7ad765a71a495



ls -ls /var/lib/lxd/containers/ubuntu-minimal
4 lrwxrwxrwx 1 root root 60 Dec 21 12:35 /var/lib/lxd/containers/ubuntu-minimal -> /var/lib/lxd/storage-pools/default/containers/ubuntu-minima

As you expected it was an permission issue:

/var/lib/lxd/storage-pools/default# ls -l
total 0
drwx------ 1 root root 110 Dec 26 05:05 containers

Changed to 755 (like fresh installation) and the containers starts again.
Thanks for having a look into the issue!

I had the same issue, which was fixed with a chmod 755 .../containers. I have no idea how the directory got in that state.

I just got /data/lxd/containers permissions reset to wrong ones on two servers by something (snap auto update?):

root@backup01 ~ # ls -l /data/lxd
total 0
drwx------ 1 root root 198 Jan 24 03:34 containers

Not sure what's causing, but it's yet another time I'm seeing it.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

srkunze picture srkunze  路  3Comments

shaun-ba picture shaun-ba  路  3Comments

sajoupa picture sajoupa  路  4Comments

mlaradji picture mlaradji  路  4Comments

jsnjack picture jsnjack  路  3Comments