Lxd: containers-snapshots: is a directory

[purell]

Required information

• Distribution: Ubuntu
• Distribution version: 18.04.2 LTS
• The output of
  
  
  
  • lxc-start --version 3.0.3
  
  
  • lxc-checkconfig
  
  

Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-4.15.0-47-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/devices
/sys/fs/cgroup/blkio
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/freezer
/sys/fs/cgroup/memory
/sys/fs/cgroup/rdma
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/pids

Cgroup v2 mount points:
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

• uname -a Linux eng-server 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• cat /proc/self/cgroup

12:pids:/user.slice/user-1001.slice/session-10.scope
11:cpuset:/
10:rdma:/
9:memory:/user.slice
8:freezer:/
7:cpu,cpuacct:/user.slice
6:blkio:/user.slice
5:devices:/user.slice
4:net_cls,net_prio:/
3:perf_event:/
2:hugetlb:/
1:name=systemd:/user.slice/user-1001.slice/session-10.scope
0::/user.slice/user-1001.slice/session-10.scope

• cat /proc/1/mounts

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=8177556k,nr_inodes=2044389,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=1641672k,mode=755 0 0
/dev/md0 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19771 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/loop0 /snap/core/6818 squashfs ro,nodev,relatime 0 0
/dev/loop1 /snap/core/6964 squashfs ro,nodev,relatime 0 0
/dev/loop2 /snap/lxd/10756 squashfs ro,nodev,relatime 0 0
/dev/loop4 /snap/lxd/10923 squashfs ro,nodev,relatime 0 0
/dev/loop3 /snap/core/6673 squashfs ro,nodev,relatime 0 0
tmpfs /run/snapd/ns tmpfs rw,nosuid,noexec,relatime,size=1641672k,mode=755 0 0
nsfs /run/snapd/ns/lxd.mnt nsfs rw 0 0
tmpfs /var/snap/lxd/common/ns tmpfs rw,relatime,size=1024k,mode=700 0 0
nsfs /var/snap/lxd/common/ns/shmounts nsfs rw 0 0
nsfs /var/snap/lxd/common/ns/mntns nsfs rw 0 0
tmpfs /run/user/1001 tmpfs rw,nosuid,nodev,relatime,size=1641668k,mode=700,uid=1001,gid=1001 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

Issue description

Any lxd command hangs. Ex:

$ lxc --debug list
DBUG[06-18|08:07:11] Connecting to a local LXD over a Unix socket
DBUG[06-18|08:07:11] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=

The system stop working around 8pm yesterday. Could this be due to an update of the core snap ?

$ snap changes
ID   Status  Spawn                   Ready                   Summary
35   Done    yesterday at 20:01 EDT  yesterday at 20:01 EDT  Auto-refresh snap "lxd"
36   Done    yesterday at 22:48 EDT  yesterday at 22:48 EDT  Running service command

Also I'm seeing the error Failed to setup shmounts, continuing without in the journal. Is it normal ?

Information to attach

• [ ] any relevant kernel output (dmesg)
• [ ] container log (The file from running lxc-start -n <c> -l <log> -o DEBUG)
• [ ] the containers configuration file
• [ ] Main daemon log (/var/snap/lxd/common/lxd/logs/lxd.log)
  t=2019-06-18T00:31:14-0400 lvl=info msg="LXD 3.14 is starting in normal mode" path=/var/snap/lxd/common/lxd t=2019-06-18T00:31:14-0400 lvl=info msg="Kernel uid/gid map:" t=2019-06-18T00:31:14-0400 lvl=info msg=" - u 0 0 4294967295" t=2019-06-18T00:31:14-0400 lvl=info msg=" - g 0 0 4294967295" t=2019-06-18T00:31:14-0400 lvl=info msg="Configured LXD uid/gid map:" t=2019-06-18T00:31:14-0400 lvl=info msg=" - u 0 1000000 1000000000" t=2019-06-18T00:31:14-0400 lvl=info msg=" - g 0 1000000 1000000000" t=2019-06-18T00:31:14-0400 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2019-06-18T00:31:14-0400 lvl=info msg="Kernel features:" t=2019-06-18T00:31:14-0400 lvl=info msg=" - netnsid-based network retrieval: no" t=2019-06-18T00:31:14-0400 lvl=info msg=" - uevent injection: no" t=2019-06-18T00:31:14-0400 lvl=info msg=" - seccomp listener: no" t=2019-06-18T00:31:14-0400 lvl=info msg=" - unprivileged file capabilities: yes" t=2019-06-18T00:31:14-0400 lvl=info msg=" - shiftfs support: no" t=2019-06-18T00:31:14-0400 lvl=info msg="Initializing local database" t=2019-06-18T00:31:14-0400 lvl=info msg="Starting /dev/lxd handler:" t=2019-06-18T00:31:14-0400 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock t=2019-06-18T00:31:14-0400 lvl=info msg="REST API daemon:" t=2019-06-18T00:31:14-0400 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket t=2019-06-18T00:31:14-0400 lvl=info msg="Initializing global database" t=2019-06-18T00:31:14-0400 lvl=info msg="Initializing storage pools" t=2019-06-18T00:31:14-0400 lvl=info msg="Applying patch: storage_api_rename_container_snapshots_dir_again" t=2019-06-18T00:31:15-0400 lvl=eror msg="Failed to start the daemon: open /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots: is a directory" t=2019-06-18T00:31:15-0400 lvl=info msg="Starting shutdown sequence" t=2019-06-18T00:31:15-0400 lvl=info msg="Stopping REST API handler:" t=2019-06-18T00:31:15-0400 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/unix.socket t=2019-06-18T00:31:15-0400 lvl=info msg="Stopping /dev/lxd handler:" t=2019-06-18T00:31:15-0400 lvl=info msg=" - closing socket" socket=/var/snap/lxd/common/lxd/devlxd/sock t=2019-06-18T00:31:15-0400 lvl=info msg="Closing the database" t=2019-06-18T00:31:15-0400 lvl=info msg="Unmounting temporary filesystems" t=2019-06-18T00:31:15-0400 lvl=info msg="Done unmounting temporary filesystems"
• [x] journalctl -u snap.lxd.daemon.service
  Jun 17 20:01:27 eng-server lxd.daemon[17703]: => Stop reason is: snap refresh Jun 17 20:01:27 eng-server lxd.daemon[17703]: => Stopping LXD Jun 17 20:01:28 eng-server systemd[1]: Stopped Service for snap application lxd.daemon. Jun 17 20:01:32 eng-server systemd[1]: Started Service for snap application lxd.daemon. Jun 17 20:01:32 eng-server lxd.daemon[17906]: => Preparing the system Jun 17 20:01:32 eng-server lxd.daemon[17906]: ==> Loading snap configuration Jun 17 20:01:32 eng-server lxd.daemon[17906]: ==> Setting up mntns symlink (mnt:[4026533408]) Jun 17 20:01:32 eng-server lxd.daemon[17906]: ==> Setting up persistent shmounts path Jun 17 20:01:33 eng-server lxd.daemon[17906]: ====> Failed to setup shmounts, continuing without Jun 17 20:01:33 eng-server lxd.daemon[17906]: ====> Making LXD shmounts use the persistent path Jun 17 20:01:33 eng-server lxd.daemon[17906]: ====> Making LXCFS use the persistent path Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Setting up kmod wrapper Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Preparing /boot Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Preparing a clean copy of /run Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Preparing a clean copy of /etc Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Setting up ceph configuration Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Setting up LVM configuration Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Rotating logs Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Setting up ZFS (0.7) Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Escaping the systemd cgroups Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Escaping the systemd process resource limits Jun 17 20:01:33 eng-server lxd.daemon[17906]: ==> Disabling shiftfs on this kernel (auto) Jun 17 20:01:33 eng-server lxd.daemon[17906]: => Starting LXCFS Jun 17 20:01:34 eng-server lxd.daemon[17906]: => Starting LXD Jun 17 20:01:34 eng-server lxd.daemon[17906]: mount namespace: 6 Jun 17 20:01:34 eng-server lxd.daemon[17906]: hierarchies: Jun 17 20:01:34 eng-server lxd.daemon[17906]: 0: fd: 7: memory Jun 17 20:01:34 eng-server lxd.daemon[17906]: 1: fd: 8: cpuset Jun 17 20:01:34 eng-server lxd.daemon[17906]: 2: fd: 9: pids Jun 17 20:01:34 eng-server lxd.daemon[17906]: 3: fd: 10: devices Jun 17 20:01:34 eng-server lxd.daemon[17906]: 4: fd: 11: freezer Jun 17 20:01:34 eng-server lxd.daemon[17906]: 5: fd: 12: net_cls,net_prio Jun 17 20:01:34 eng-server lxd.daemon[17906]: 6: fd: 13: hugetlb Jun 17 20:01:34 eng-server lxd.daemon[17906]: 7: fd: 14: rdma Jun 17 20:01:34 eng-server lxd.daemon[17906]: 8: fd: 15: perf_event Jun 17 20:01:34 eng-server lxd.daemon[17906]: 9: fd: 16: cpu,cpuacct Jun 17 20:01:34 eng-server lxd.daemon[17906]: 10: fd: 17: blkio Jun 17 20:01:34 eng-server lxd.daemon[17906]: 11: fd: 18: name=systemd Jun 17 20:01:34 eng-server lxd.daemon[17906]: 12: fd: 19: unified Jun 17 20:01:34 eng-server lxd.daemon[17906]: t=2019-06-17T20:01:34-0400 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." Jun 17 20:01:34 eng-server lxd.daemon[17906]: t=2019-06-17T20:01:34-0400 lvl=eror msg="Failed to start the daemon: open /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots: is a directory"
• [x] systemctl | grep lxd
  run-snapd-ns-lxd.mnt.mount loaded active mounted /run/snapd/ns/lxd.mnt snap-lxd-10756.mount loaded active mounted Mount unit for lxd, revision 10756 snap-lxd-10923.mount loaded active mounted Mount unit for lxd, revision 10923 var-snap-lxd-common-ns-mntns.mount loaded active mounted /var/snap/lxd/common/ns/mntns var-snap-lxd-common-ns-shmounts.mount loaded active mounted /var/snap/lxd/common/ns/shmounts var-snap-lxd-common-ns.mount loaded active mounted /var/snap/lxd/common/ns ● lxd-host-dns.service loaded failed failed LXD host DNS service ● snap.lxd.daemon.service loaded failed failed Service for snap application lxd.daemon ● snap.lxd.daemon.unix.socket loaded failed failed Socket unix for snap application lxd.daemon

[stgraber]

So this got tracked down, I'm now working on setting up a system with that issue so I can prepare a proper fix for it.

[AquaroTorres]

Same problem

[AquaroTorres]

Sorry for my ignorance, but how can I apply this patch to my current lxd installation?

[stgraber]

@AquaroTorres if you're using the snap, we will push this and another cherry-picked fix later today.
Until then your best bet is to force LXD to just skip this patch by creating a /var/snap/lxd/common/lxd/database/patch.local.sql file containing:

INSERT INTO patches (name, applied_at) VALUES ('storage_api_rename_container_snapshots_dir_again', '2019-06-18T14:10:59Z');