Lxd: Snap LXD 4.6 does not start - upgrade error for cluster database (instance is not in a cluster).
Required information
- Distribution:Ubuntu
- Distribution version: 18.04
- The output of "lxc info" or if that fails:
- Kernel version: 4.15.0-118-generic
- LXC version: 4.6
- LXD version: 4.6
- Storage backend in use: zfs
Issue description
Server maintenance and 'lxc list' returned Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory". Error searches return similar results but dated years prior for solution and most seemed to be resolved via tarball to @stgraber. snap refresh and hello-world installs complete with no errors. apt lxd is/was not installed.....
Steps to reproduce
UNKNOWN - discovered problem after maintenance window
Information to attach
After running the following from other posts -
On one of the affected systems, please do (as root):
systemctl stop snap.lxd.daemon snap.lxd.daemon.unix.socket &
pkill -9 -f lxd.*logf
lxd --debug --group lxd
And post the output here.
Output -
root@xpopl-kvmlxd-52:~# lxc list
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory
root@xpopl-kvmlxd-52:~# systemctl stop snap.lxd.daemon snap.lxd.daemon.unix.socket &
[1] 28819
root@xpopl-kvmlxd-52:~# pkill -9 -f lxd.*logf
[1]+ Done systemctl stop snap.lxd.daemon snap.lxd.daemon.unix.socket
root@xpopl-kvmlxd-52:~# lxd --debug --group lxd
INFO[09-22|18:31:52] LXD 4.6 is starting in normal mode path=/var/snap/lxd/common/lxd
INFO[09-22|18:31:52] Kernel uid/gid map:
INFO[09-22|18:31:52] - u 0 0 4294967295
INFO[09-22|18:31:52] - g 0 0 4294967295
INFO[09-22|18:31:52] Configured LXD uid/gid map:
INFO[09-22|18:31:52] - u 0 1000000 1000000000
INFO[09-22|18:31:52] - g 0 1000000 1000000000
INFO[09-22|18:31:52] Kernel features:
INFO[09-22|18:31:52] - closing multiple file descriptors efficiently: no
INFO[09-22|18:31:52] - netnsid-based network retrieval: no
INFO[09-22|18:31:52] - pidfds: no
INFO[09-22|18:31:52] - uevent injection: no
INFO[09-22|18:31:52] - seccomp listener: no
INFO[09-22|18:31:52] - seccomp listener continue syscalls: no
INFO[09-22|18:31:52] - seccomp listener add file descriptors: no
INFO[09-22|18:31:52] - attach to namespaces via pidfds: no
INFO[09-22|18:31:52] - safe native terminal allocation : no
INFO[09-22|18:31:52] - unprivileged file capabilities: yes
INFO[09-22|18:31:52] - cgroup layout: hybrid
WARN[09-22|18:31:52] - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
INFO[09-22|18:31:52] - shiftfs support: no
INFO[09-22|18:31:52] Initializing local database
DBUG[09-22|18:31:52] Initializing database gateway
DBUG[09-22|18:31:52] Start database node id=1 address= role=voter
INFO[09-22|18:31:52] Starting /dev/lxd handler:
INFO[09-22|18:31:52] - binding devlxd socket socket=/var/snap/lxd/common/lxd/devlxd/sock
INFO[09-22|18:31:52] REST API daemon:
INFO[09-22|18:31:52] - binding Unix socket socket=/var/snap/lxd/common/lxd/unix.socket
INFO[09-22|18:31:52] - binding TCP socket socket=[::]:8443
INFO[09-22|18:31:52] Initializing global database
DBUG[09-22|18:31:52] Dqlite: attempt 0: server 1: connected
INFO[09-22|18:31:52] Updating the LXD global schema. Backup made as "global.bak"
DBUG[09-22|18:31:53] Updating global DB schema from 33 to 34
DBUG[09-22|18:31:53] Updating global DB schema from 34 to 35
DBUG[09-22|18:31:53] Database error: &errors.errorString{s:"failed to apply update 34: invalid ID"}
EROR[09-22|18:31:53] Failed to start the daemon: failed to open cluster database: failed to ensure schema: failed to apply update 34: invalid ID
INFO[09-22|18:31:53] Starting shutdown sequence
INFO[09-22|18:31:53] Stop database gateway
INFO[09-22|18:31:53] Stopping REST API handler:
INFO[09-22|18:31:53] - closing socket socket=[::]:8443
INFO[09-22|18:31:53] - closing socket socket=/var/snap/lxd/common/lxd/unix.socket
INFO[09-22|18:31:53] Stopping /dev/lxd handler:
INFO[09-22|18:31:53] - closing socket socket=/var/snap/lxd/common/lxd/devlxd/sock
DBUG[09-22|18:31:53] Not unmounting temporary filesystems (containers are still running)
Error: failed to open cluster database: failed to ensure schema: failed to apply update 34: invalid ID
cawhite6
All 32 comments
Ok, so you're upgrading from something a bit older right? Like 4.2 or 4.3?
This particular patch should not trigger during a normal 4.5 to 4.6 or even 4.4 to 4.6.
In any case, this is the patch that dedups entries in storage_volumes for users of ceph/cephfs.
Are you using ceph or cephfs?
And could you send me a tarball of /var/snap/lxd/common/lxd/database to stgraber at ubuntu dot com?
The most likely source is bad/inconsistent volume records in the current DB which are tripping the upgrade.
If that's the case, I can provide you with a patch.sql file to fix them, unblocking the upgrade.
If it's broken upgrade logic, we'll sort it out quickly too.
stgraber
on 22 Sep 2020
Thanks for the quick reply - I've sent over the tarball. I am not sure what upgrade was previous - this is a secondary server and our primary is running fine - identical installation, same 4.6 snap version.
Both using zfs filesystem on containers
cawhite6
on 22 Sep 2020
@cawhite6 you indeed do have an inconsistent DB record from a previous update.
The id of storage_volumes and storage_volumes_snapshots must be unique to allow for an aggregate view to work properly.
To fix your setup, write a file at database/patch.global.sql containing:
UPDATE storage_volumes_snapshots SET id=191 WHERE id=6;
Then start LXD again and it will succeed the upgrade as seen here upgrading using the DB you sent me:
INFO[09-22|15:22:14] Initializing global database
DBUG[09-22|15:22:14] Dqlite: attempt 0: server 1: connected
INFO[09-22|15:22:14] Updating the LXD global schema. Backup made as "global.bak"
DBUG[09-22|15:22:14] Running pre-update queries from file for global DB schema
DBUG[09-22|15:22:14] Updating global DB schema from 33 to 34
DBUG[09-22|15:22:14] Updating global DB schema from 34 to 35
DBUG[09-22|15:22:14] Updating global DB schema from 35 to 36
DBUG[09-22|15:22:14] Updating global DB schema from 36 to 37
DBUG[09-22|15:22:14] Updating global DB schema from 37 to 38
INFO[09-22|15:22:16] Firewall loaded driver "nftables"
For anyone else attempting this kind of fix on their own, be very careful.
You can see if there's something wrong by running SELECT * FROM storage_volumes_snapshots WHERE id IN (SELECT id FROM storage_volumes);
If you find a record, you'll need to update its id to a value which has not yet been used in either table.
stgraber
on 22 Sep 2020
@cawhite6 you indeed do have an inconsistent DB record from a previous update.
Theidofstorage_volumesandstorage_volumes_snapshotsmust be unique to allow for an aggregate view to work properly.To fix your setup, write a file at
database/patch.global.sqlcontaining:UPDATE storage_volumes_snapshots SET id=191 WHERE id=6;Then start LXD again and it will succeed the upgrade as seen here upgrading using the DB you sent me:
INFO[09-22|15:22:14] Initializing global database DBUG[09-22|15:22:14] Dqlite: attempt 0: server 1: connected INFO[09-22|15:22:14] Updating the LXD global schema. Backup made as "global.bak" DBUG[09-22|15:22:14] Running pre-update queries from file for global DB schema DBUG[09-22|15:22:14] Updating global DB schema from 33 to 34 DBUG[09-22|15:22:14] Updating global DB schema from 34 to 35 DBUG[09-22|15:22:14] Updating global DB schema from 35 to 36 DBUG[09-22|15:22:14] Updating global DB schema from 36 to 37 DBUG[09-22|15:22:14] Updating global DB schema from 37 to 38 INFO[09-22|15:22:16] Firewall loaded driver "nftables"
Perfect - worked like a charm, back up and running (I feared the worst)!
cawhite6
on 22 Sep 2020
Same issue with snap package. lxd isn't starting now
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3808]: => LXD failed to start
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 91.
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: Stopped Service for snap application lxd.daemon.
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal systemd[1]: Started Service for snap application lxd.daemon.
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: => Preparing the system (17320)
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Loading snap configuration
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Setting up mntns symlink (mnt:[4026532401])
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Setting up kmod wrapper
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Preparing /boot
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Preparing a clean copy of /run
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Preparing /run/bin
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Preparing a clean copy of /etc
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Preparing a clean copy of /usr/share/misc
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Setting up ceph configuration
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Setting up LVM configuration
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Rotating logs
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Setting up ZFS (0.7)
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Escaping the systemd cgroups
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ====> Detected cgroup V1
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Escaping the systemd process resource limits
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: ==> Disabling shiftfs on this kernel (auto)
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: Closed liblxcfs.so
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: Running destructor lxcfs_exit
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: Running constructor lxcfs_init to reload liblxcfs
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal kernel: [ 1206.009065] new mount options do not match the existing superblock, will be ignored
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: mount namespace: 5
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: hierarchies:
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 0: fd: 6:
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 1: fd: 7: name=systemd
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 2: fd: 8: cpu,cpuacct
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 3: fd: 9: net_cls,net_prio
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 4: fd: 10: hugetlb
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 5: fd: 11: pids
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 6: fd: 12: blkio
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 7: fd: 13: cpuset
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 8: fd: 14: memory
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 9: fd: 15: rdma
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 10: fd: 16: perf_event
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 11: fd: 17: freezer
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: 12: fd: 19: devices
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: Kernel does not support swap accounting
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: api_extensions:
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - cgroups
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - sys_cpu_online
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_cpuinfo
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_diskstats
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_loadavg
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_meminfo
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_stat
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_swaps
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - proc_uptime
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - shared_pidns
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - cpuview_daemon
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - loadavg_daemon
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: - pidfds
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[14339]: Reloaded LXCFS
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: => Re-using existing LXCFS
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: => Starting LXD
Sep 24 08:09:17 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: t=2020-09-24T08:09:17+0200 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
Sep 24 08:09:19 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: t=2020-09-24T08:09:19+0200 lvl=eror msg="Failed to start the daemon: failed to open cluster database: failed to ensure schema: failed to apply update 34: invalid ID"
Sep 24 08:09:19 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: Error: failed to open cluster database: failed to ensure schema: failed to apply update 34: invalid ID
Sep 24 08:09:19 Ubuntu-1804-bionic-64-minimal lxd.daemon[3964]: => LXD failed to start
Sep 24 08:09:19 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 08:09:19 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Sep 24 08:09:20 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Sep 24 08:09:20 Ubuntu-1804-bionic-64-minimal systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 92.
iamhermes
on 24 Sep 2020
Can you tar up /var/snap/lxd/common/lxd/database and send to thomas dot parrott at canonical dot com?
tomponline
on 24 Sep 2020
link for the database tar (127MB) is sent
iamhermes
on 24 Sep 2020
@iamhermes received thanks.
Running the query @stgraber mentioned above, shows an inconsistency:
sqlite> SELECT * FROM storage_volumes_snapshots WHERE id IN (SELECT id FROM storage_volumes);
21501|4111|snap4419||
sqlite> select MAX(id) from storage_volumes_snapshots;
32534
sqlite> select MAX(id) from storage_volumes;
21501
So I think a patch query should fix it:
UPDATE storage_volumes_snapshots SET id=32535 WHERE id=21501;
But I am not certain as to the relationship between IDs in these tables, and whether just selecting the max snapshot ID +1 would prevent future issues, so would like confirmation from @stgraber first.
tomponline
on 24 Sep 2020
@tomponline it must be an unused id in both tables, usually I would pick the next id from snapshots and just make sure it's not already used for a volume.
stgraber
on 24 Sep 2020
@stgraber thanks, so @iamhermes this should fix it (as ID 32535 is not used in either table) :
UPDATE storage_volumes_snapshots SET id=32535 WHERE id=21501;
Problem solved. Thanks. All containers are up again
iamhermes
on 24 Sep 2020
For other readers affected by this issue, in order to be able to run the query SELECT * FROM storage_volumes_snapshots WHERE id IN (SELECT id FROM storage_volumes); on an LXD that will not start, you need to use the sqlite3 command to open the database read only DB file as so:
sqlite3 /var/snap/lxd/common/lxd/database/global/db.bin
sqlite> SELECT * FROM storage_volumes_snapshots WHERE id IN (SELECT id FROM storage_volumes);
This way you can open the database without needing to start LXD.
tomponline
on 5 Oct 2020
@tomponline Thanks for this, so I run the SQL, got this.
sqlite> SELECT * FROM storage_volumes_snapshots WHERE id IN (SELECT id FROM storage_volumes);
262|258|01-26-20-01-23||
264|258|02-02-20-01-23||
sqlite> select MAX(id) from storage_volumes_snapshots;
421
sqlite> select MAX(id) from storage_volumes;
480
sqlite>
So I should do this then? But I am confused about that I got 2 results from the first Select from.
UPDATE storage_volumes_snapshots SET id=422 WHERE id=480;
dkruyt
on 5 Oct 2020
@dkruyt as you have to affected volumes you would need 2 update queries, 1 for each volume, setting a unique ID for each one.
You need to create a patch file as mentioned in https://github.com/lxc/lxd/issues/7920#issuecomment-696930106
tomponline
on 5 Oct 2020
@tomponline , ok created the patch file and restarted, now it is running,
UPDATE storage_volumes_snapshots SET id=481 WHERE id=262;
UPDATE storage_volumes_snapshots SET id=482 WHERE id=264;
But, still got the socket error...
lxc list
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
lxd seems running...
systemctl status snap.lxd.daemon.service
● snap.lxd.daemon.service - Service for snap application lxd.daemon
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
Active: active (running) since Mon 2020-10-05 11:31:47 CEST; 2min 46s ago
Process: 1289 ExecStop=/usr/bin/snap run --command=stop lxd.daemon (code=exited, status=0/SUCCESS)
Process: 28001 ExecReload=/usr/bin/snap run --command=reload lxd.daemon (code=exited, status=0/SUCCESS)
Main PID: 2486 (daemon.start)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/snap.lxd.daemon.service
‣ 2486 /bin/sh /snap/lxd/17497/commands/daemon.start
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up LVM configuration
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Rotating logs
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up ZFS (0.8)
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Escaping the systemd cgroups
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ====> Detected cgroup V1
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Escaping the systemd process resource limits
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Enabling CRIU
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Disabling shiftfs on this kernel (auto)
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: => Starting LXCFS
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: Running constructor lxcfs_init to reload liblxcfs
Oct 05 11:31:47 lxdh02.klingon.nl systemd[1]: Started Service for snap application lxd.daemon.
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: => Preparing the system (17497)
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Loading snap configuration
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up mntns symlink (mnt:[4026532598])
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up kmod wrapper
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Preparing /boot
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Preparing a clean copy of /run
Oct 05 11:31:47 lxdh02.klingon.nl lxd.daemon[2486]: ==> Preparing /run/bin
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Preparing a clean copy of /etc
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Preparing a clean copy of /usr/share/misc
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up ceph configuration
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up LVM configuration
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Rotating logs
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Setting up ZFS (0.8)
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Escaping the systemd cgroups
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ====> Detected cgroup V1
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Escaping the systemd process resource limits
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Enabling CRIU
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: ==> Disabling shiftfs on this kernel (auto)
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: => Starting LXCFS
Oct 05 11:31:48 lxdh02.klingon.nl lxd.daemon[2486]: Running constructor lxcfs_init to reload liblxcfs
ls -la /var/snap/lxd/common/lxd/unix.socket
srw-rw---- 1 root root 0 Oct 5 11:32 /var/snap/lxd/common/lxd/unix.socket
dkruyt
on 5 Oct 2020
Can you show output of ps aux | grep lxd please
tomponline
on 5 Oct 2020
root 19883 0.0 0.0 4636 1684 ? Ss 11:50 0:00 /bin/sh /snap/lxd/17497/commands/daemon.start
root 20039 3.9 0.0 97804 1676 ? Sl 11:50 0:08 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root 20046 9.1 2.1 1857912 176180 ? SLl 11:50 0:19 lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd
root 20047 0.1 0.4 1303136 33956 ? SLl 11:50 0:00 lxd waitready
root 20048 0.0 0.0 4636 976 ? S 11:50 0:00 /bin/sh /snap/lxd/17497/commands/daemon.start
lxd 20218 0.3 0.0 49964 2692 ? Ss 11:51 0:00 dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.20.216.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.20.216.2,10.20.216.254,1h --listen-address=fd42:7a27:d82c:2f6b::1 --enable-ra --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd
From: /var/snap/lxd/common/lxd/logs/lxd.log
t=2020-10-05T11:51:06+0200 lvl=info msg="Starting container" action=start created=2020-03-17T08:17:29+0100 ephemeral=false name=bitwarden-rs project=default stateful=false used=2020-03-17T08:17:54+0100
t=2020-10-05T11:51:06+0200 lvl=info msg="Downloading image" alias=18.04 server=https://cloud-images.ubuntu.com/minimal/releases
t=2020-10-05T11:51:06+0200 lvl=eror msg="Failed starting container" action=start created=2020-03-17T08:17:29+0100 ephemeral=false name=bitwarden-rs project=default stateful=false used=2020-03-17T08:17:54+0100
t=2020-10-05T11:51:06+0200 lvl=eror msg="Failed to start instance 'bitwarden-rs': Failed to run: /snap/lxd/current/bin/lxd forkstart bitwarden-rs /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/bitwarden-rs/lxc.conf: "
t=2020-10-05T11:51:07+0200 lvl=info msg="Starting container" action=start created=2020-03-15T22:36:23+0100 ephemeral=false name=minecraft project=default stateful=false used=2020-08-18T17:54:30+0200
t=2020-10-05T11:51:07+0200 lvl=eror msg="Failed starting container" action=start created=2020-03-15T22:36:23+0100 ephemeral=false name=minecraft project=default stateful=false used=2020-08-18T17:54:30+0200
t=2020-10-05T11:51:07+0200 lvl=eror msg="Failed to start instance 'minecraft': Failed to run: /snap/lxd/current/bin/lxd forkstart minecraft /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/minecraft/lxc.conf: "
t=2020-10-05T11:51:07+0200 lvl=info msg="Starting container" action=start created=2019-08-14T00:35:16+0200 ephemeral=false name=mx2 project=default stateful=false used=2020-01-17T14:27:46+0100
t=2020-10-05T11:51:08+0200 lvl=eror msg="Failed starting container" action=start created=2019-08-14T00:35:16+0200 ephemeral=false name=mx2 project=default stateful=false used=2020-01-17T14:27:46+0100
t=2020-10-05T11:51:08+0200 lvl=eror msg="Failed to start instance 'mx2': Failed to run: /snap/lxd/current/bin/lxd forkstart mx2 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/mx2/lxc.conf: "
t=2020-10-05T11:51:08+0200 lvl=info msg="Starting container" action=start created=2019-08-14T00:34:28+0200 ephemeral=false name=ns2 project=default stateful=false used=2020-01-17T14:27:47+0100
t=2020-10-05T11:51:09+0200 lvl=eror msg="Failed starting container" action=start created=2019-08-14T00:34:28+0200 ephemeral=false name=ns2 project=default stateful=false used=2020-01-17T14:27:47+0100
t=2020-10-05T11:51:09+0200 lvl=eror msg="Failed to start instance 'ns2': Failed to run: /snap/lxd/current/bin/lxd forkstart ns2 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/ns2/lxc.conf: "
t=2020-10-05T11:51:10+0200 lvl=info msg="Starting container" action=start created=2019-10-06T19:12:15+0200 ephemeral=false name=revproxy02 project=default stateful=false used=2020-03-17T08:39:10+0100
t=2020-10-05T11:51:11+0200 lvl=eror msg="Failed starting container" action=start created=2019-10-06T19:12:15+0200 ephemeral=false name=revproxy02 project=default stateful=false used=2020-03-17T08:39:10+0100
t=2020-10-05T11:51:11+0200 lvl=eror msg="Failed to start instance 'revproxy02': Failed to run: /snap/lxd/current/bin/lxd forkstart revproxy02 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/revproxy02/lxc.conf: "
t=2020-10-05T11:51:11+0200 lvl=warn msg="Deleting leftover/partially unpacked image volume" driver=zfs fingerprint=9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790 pool=lxdpool01
t=2020-10-05T11:51:21+0200 lvl=info msg="Image downloaded" alias=18.04 server=https://cloud-images.ubuntu.com/minimal/releases
t=2020-10-05T11:51:21+0200 lvl=info msg="Downloading image" alias=18.04 server=https://cloud-images.ubuntu.com/releases
t=2020-10-05T11:51:30+0200 lvl=warn msg="Deleting leftover/partially unpacked image volume" driver=zfs fingerprint=39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810 pool=lxdpool01
t=2020-10-05T11:51:45+0200 lvl=info msg="Image downloaded" alias=18.04 server=https://cloud-images.ubuntu.com/releases
t=2020-10-05T11:51:46+0200 lvl=info msg="Downloading image" alias=20.04 server=https://cloud-images.ubuntu.com/releases
t=2020-10-05T11:52:14+0200 lvl=info msg="Image downloaded" alias=20.04 server=https://cloud-images.ubuntu.com/releases
dkruyt
on 5 Oct 2020
So so as you say, lxd seems to be starting now, but you've got other issues going on here.
Can you see any of the container specific logs in /var/snap/lxd/common/lxd/logs/?
tomponline
on 5 Oct 2020
Container logs looks like this...
lxc ns2 20201005095109.195 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc ns2 20201005095109.195 ERROR start - start.c:lxc_init:798 - Failed to run lxc.hook.pre-start for container "ns2"
lxc ns2 20201005095109.195 ERROR start - start.c:__lxc_start:1945 - Failed to initialize container "ns2"
lxc ns2 20201005095109.274 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc ns2 20201005095109.274 ERROR start - start.c:lxc_end:923 - Failed to run "lxc.hook.stop" hook
lxc ns2 20201005095109.961 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc ns2 20201005095109.962 ERROR start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "ns2"
lxc ns2 20201005095109.962 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:841 - No such file or directory - Failed to receive the container state
Can you run:
snap set lxd daemon.debug=true
systemctl reload snap.lxd.daemon
tail -f /var/snap/lxd/common/lxd/logs/lxd.log
I think it has todo with this error, something in the database...
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Database error: &errors.errorString{s:\"No such object\"}"
t=2020-10-05T12:56:31+0200 lvl=eror msg="Failed to start instance 'ns2': Failed to run: /snap/lxd/current/bin/lxd forkstart ns2 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/ns2/lxc.conf: "
t=2020-10-05T12:56:31+0200 lvl=dbug msg="Scheduler: network: veth29f25ad3 has been added: updating network priorities"
t=2020-10-05T12:56:31+0200 lvl=dbug msg="Scheduler: network: vethabc8372d has been added: updating network priorities"
t=2020-10-05T12:56:31+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=dbug msg="UpdateInstanceBackupFile started" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=dbug msg="UpdateInstanceBackupFile finished" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=dbug msg="MountInstance started" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=dbug msg="MountInstance finished" driver=zfs instance=revproxy02 pool=lxdpool01 project=default
t=2020-10-05T12:56:31+0200 lvl=info msg="Starting container" action=start created=2019-10-06T19:12:15+0200 ephemeral=false name=revproxy02 project=default stateful=false used=2020-03-17T08:39:10+0100
t=2020-10-05T12:56:32+0200 lvl=eror msg="Failed starting container" action=start created=2019-10-06T19:12:15+0200 ephemeral=false name=revproxy02 project=default stateful=false used=2020-03-17T08:39:10+0100
t=2020-10-05T12:56:32+0200 lvl=eror msg="Failed to start instance 'revproxy02': Failed to run: /snap/lxd/current/bin/lxd forkstart revproxy02 /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/revproxy02/lxc.conf: "
t=2020-10-05T12:56:45+0200 lvl=dbug msg="EnsureImage started" driver=zfs fingerprint=4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:56:45+0200 lvl=dbug msg="Database error: &errors.errorString{s:\"No such object\"}"
t=2020-10-05T12:56:45+0200 lvl=warn msg="Deleting leftover/partially unpacked image volume" driver=zfs fingerprint=4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:56:46+0200 lvl=dbug msg="Mounted ZFS dataset" dev=lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:56:46+0200 lvl=dbug msg="Running filler function" dev= driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Unmounted ZFS dataset" dev=lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c driver=zfs path=/var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Database error: &errors.errorString{s:\"No such object\"}"
t=2020-10-05T12:57:05+0200 lvl=dbug msg="EnsureImage finished" driver=zfs fingerprint=4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c pool=lxdpool01
t=2020-10-05T12:57:05+0200 lvl=info msg="Image downloaded" alias=20.04 server=https://cloud-images.ubuntu.com/releases
t=2020-10-05T12:57:05+0200 lvl=dbug msg="DeleteImage started" driver=zfs fingerprint=97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540 pool=lxdpool01
t=2020-10-05T12:57:05+0200 lvl=dbug msg="DeleteImage finished" driver=zfs fingerprint=97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540 pool=lxdpool01
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Processing image" alias=18.04 fp=9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790 protocol=simplestreams server=https://cloud-images.ubuntu.com/minimal/releases
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Connecting to a remote simplestreams server"
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Image \"9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790\" already exists in the DB"
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Image already exists on storage pool \"lxdpool01\""
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Already up to date" fp=9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790
t=2020-10-05T12:57:05+0200 lvl=dbug msg="Success for task operation: 3b4af87a-18a0-4e81-8c9f-ffdd008e24e3"
@dkruyt not necessarily, but perhaps, that is (in my opinion) a rather alarming message that can occur in a normal scenario when a DB row is looked up and not found. It occurs quite often in my experience and doesn't normally indicate an issue.
Please can you describe more about what version of lxd you were running before the upgrade, and what storage pool configure you had. Thanks
tomponline
on 5 Oct 2020
The previous versie would be then 4.4 or 4.5, its just updating with snap.
> snap list lxd
Name Version Rev Tracking Publisher Notes
lxd 4.6 17497 latest/stable canonical✓ -
> lxd --version
4.6
> lxc --version
4.6
Storagepool is ZFS.
zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
lxdpool01 179G 75.1G 104G - 44% 41% 1.00x ONLINE -
dkruyt
on 5 Oct 2020
@stgraber do you have any ideas for this, lxd seems to be starting now after the DB patch, but containers not starting.
tomponline
on 5 Oct 2020
Can you show zfs list -t all? Error would be consistent with an incorrect mountpoint.
stgraber
on 5 Oct 2020
Can you show
zfs list -t all? Error would be consistent with an incorrect mountpoint.
@stgraber
NAME USED AVAIL REFER MOUNTPOINT
lxdpool01 75.1G 98.3G 24K none
lxdpool01/containers 67.3G 98.3G 24K none
lxdpool01/containers/at-zt-demo02 293M 98.3G 409M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/at-zt-demo02
lxdpool01/containers/bionic-minimal 650M 98.3G 650M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/bionic-minimal
lxdpool01/containers/bitwarden 31.5G 98.3G 15.9G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/bitwarden
lxdpool01/containers/bitwarden@deleted-cc42e4e0-fd3d-4dc4-b939-e802a9bca099 77.8M - 13.3G -
lxdpool01/containers/bitwarden@snapshot-01-26-20-01-23 68.7M - 17.2G -
lxdpool01/containers/bitwarden@snapshot-02-02-20-01-23 69.3M - 17.2G -
lxdpool01/containers/bitwarden@snapshot-02-07-20-22-23 57.0M - 17.3G -
lxdpool01/containers/bitwarden@snapshot-02-09-20-01-23 69.3M - 17.3G -
lxdpool01/containers/bitwarden-rs 2.18G 98.3G 1.92G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/bitwarden-rs
lxdpool01/containers/bitwarden-rs@snapshot-snap0 222M - 1.10G -
lxdpool01/containers/bitwarden-rs@snapshot-snap1 205M - 1.49G -
lxdpool01/containers/centos 45.7M 98.3G 253M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/centos
lxdpool01/containers/docker-learn 1.05G 98.3G 1.51G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/docker-learn
lxdpool01/containers/docker-template 155K 98.3G 675M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/docker-template
lxdpool01/containers/docker-template@copy-ee89e692-8a96-487b-8c28-bc8ea7b47d8e 1K - 675M -
lxdpool01/containers/docker01 2.45G 98.3G 2.38G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/docker01
lxdpool01/containers/docker01@copy-e587da0c-c3b9-4c53-94eb-af322210d799 55.1M - 675M -
lxdpool01/containers/docker01@copy-53820025-1962-42d4-8d26-c40ead5337ad 59.4M - 936M -
lxdpool01/containers/docker01@snapshot-02-10-20-16-14 62.9M - 2.38G -
lxdpool01/containers/ghost-02 3.71G 98.3G 3.81G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/ghost-02
lxdpool01/containers/ghost-dev 586M 98.3G 802M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/ghost-dev
lxdpool01/containers/ghostblog-restore 106M 98.3G 2.79G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/ghostblog-restore
lxdpool01/containers/mailserver-testupgrade 1.99G 98.3G 1.99G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/mailserver-testupgrade
lxdpool01/containers/minecraft 423M 98.3G 527M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/minecraft
lxdpool01/containers/mx2 1.00G 98.3G 1.07G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/mx2
lxdpool01/containers/ns1-restored 5.08M 98.3G 352M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/ns1-restored
lxdpool01/containers/ns2 316M 98.3G 404M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/ns2
lxdpool01/containers/revproxy-old 162M 98.3G 303M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/revproxy-old
lxdpool01/containers/revproxy-restore 165M 98.3G 706M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/revproxy-restore
lxdpool01/containers/revproxy02 806M 98.3G 806M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/revproxy02
lxdpool01/containers/shinobi 912M 98.3G 1.23G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/shinobi
lxdpool01/containers/unms 13.7G 98.3G 14.0G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/unms
lxdpool01/containers/zoneminder 5.24G 98.3G 5.74G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers/zoneminder
lxdpool01/custom 24K 98.3G 24K none
lxdpool01/custom-snapshots 24K 98.3G 24K none
lxdpool01/deleted 6.67G 98.3G 24K none
lxdpool01/deleted/containers 24K 98.3G 24K none
lxdpool01/deleted/custom 24K 98.3G 24K none
lxdpool01/deleted/images 6.67G 98.3G 24K none
lxdpool01/deleted/images/079e64c12313e2a0d747e3914028f8115e756e461b12926625cefa4a376b54e8 181M 98.3G 181M none
lxdpool01/deleted/images/079e64c12313e2a0d747e3914028f8115e756e461b12926625cefa4a376b54e8@readonly 0B - 181M -
lxdpool01/deleted/images/208d20a60acbdc1e76eb384900d2b03c1857002c3c89d7581eab07a1fdd2f799 181M 98.3G 181M none
lxdpool01/deleted/images/208d20a60acbdc1e76eb384900d2b03c1857002c3c89d7581eab07a1fdd2f799@readonly 0B - 181M -
lxdpool01/deleted/images/2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac 340M 98.3G 340M none
lxdpool01/deleted/images/2dd611e2689a8efc45807bd2a86933cf2da0ffc768f57814724a73b5db499eac@readonly 0B - 340M -
lxdpool01/deleted/images/36f4ed21a7f832dc8f9de0825970f4cfa570ce171d2524e090cb4d35d96e7471 213M 98.3G 213M none
lxdpool01/deleted/images/36f4ed21a7f832dc8f9de0825970f4cfa570ce171d2524e090cb4d35d96e7471@readonly 0B - 213M -
lxdpool01/deleted/images/39b5bbd7cf7e3642631d463c1e8582bfb50a1b9cc4c792026e6791c5e5dde60d 671M 98.3G 671M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/39b5bbd7cf7e3642631d463c1e8582bfb50a1b9cc4c792026e6791c5e5dde60d
lxdpool01/deleted/images/39b5bbd7cf7e3642631d463c1e8582bfb50a1b9cc4c792026e6791c5e5dde60d@readonly 0B - 671M -
lxdpool01/deleted/images/7fa200f4dfbe64189734eedf7b2d8b4e7d304f840a6ef4dc01afa0e005f9396f 2.79G 98.3G 2.79G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/7fa200f4dfbe64189734eedf7b2d8b4e7d304f840a6ef4dc01afa0e005f9396f
lxdpool01/deleted/images/7fa200f4dfbe64189734eedf7b2d8b4e7d304f840a6ef4dc01afa0e005f9396f@readonly 0B - 2.79G -
lxdpool01/deleted/images/8c4e87e53c024e0449003350f0b0626b124b68060b73c0a7ad9547670e00d4b3 342M 98.3G 342M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/8c4e87e53c024e0449003350f0b0626b124b68060b73c0a7ad9547670e00d4b3
lxdpool01/deleted/images/8c4e87e53c024e0449003350f0b0626b124b68060b73c0a7ad9547670e00d4b3@readonly 0B - 342M -
lxdpool01/deleted/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185 342M 98.3G 342M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185
lxdpool01/deleted/images/979ff60086ca9fb1c93d3131f8eca291820a524f0bd0b7d621a3c2f5f41ef185@readonly 0B - 342M -
lxdpool01/deleted/images/97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540 533M 98.3G 533M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540
lxdpool01/deleted/images/97c470e427c425cf2ec4d7d55b6f1397ea55043c518b194a58fc6b9da426f540@readonly 0B - 533M -
lxdpool01/deleted/images/a7cee2f531e76a81e6d7efefaf96cb4804346cfbc45e9497a7701247bb648c27 449M 98.3G 449M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/a7cee2f531e76a81e6d7efefaf96cb4804346cfbc45e9497a7701247bb648c27
lxdpool01/deleted/images/a7cee2f531e76a81e6d7efefaf96cb4804346cfbc45e9497a7701247bb648c27@readonly 0B - 449M -
lxdpool01/deleted/images/a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6 358M 98.3G 358M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6
lxdpool01/deleted/images/a92eaa65a5c5e53c6bf788b4443f4e5d2afac1665486247c336aa90959522bb6@readonly 0B - 358M -
lxdpool01/deleted/images/e9000af2a59f3ea189fba9dddd283f502b7ceca09bca5d5af64fae9f8c0eb4e9 182M 98.3G 182M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/e9000af2a59f3ea189fba9dddd283f502b7ceca09bca5d5af64fae9f8c0eb4e9
lxdpool01/deleted/images/e9000af2a59f3ea189fba9dddd283f502b7ceca09bca5d5af64fae9f8c0eb4e9@readonly 0B - 182M -
lxdpool01/deleted/images/f46afc16f1e9e2f94691466fc6de3f179ac408ab6470fd1f988d420eb0cbf7e4 182M 98.3G 182M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/f46afc16f1e9e2f94691466fc6de3f179ac408ab6470fd1f988d420eb0cbf7e4
lxdpool01/deleted/images/f46afc16f1e9e2f94691466fc6de3f179ac408ab6470fd1f988d420eb0cbf7e4@readonly 0B - 182M -
lxdpool01/deleted/virtual-machines 24K 98.3G 24K none
lxdpool01/images 1.06G 98.3G 24K none
lxdpool01/images/39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810 359M 98.3G 359M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810
lxdpool01/images/39a93d0b355279d430e8ce21c689aa88515212ee99874276e77f7f31ad7bf810@readonly 0B - 359M -
lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c 533M 98.3G 533M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c
lxdpool01/images/4746a4889a31e449e6c5e2764eb733a60e85131be3165c91d5f5a8346f0ba43c@readonly 0B - 533M -
lxdpool01/images/9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790 197M 98.3G 197M /var/snap/lxd/common/lxd/storage-pools/lxdpool01/images/9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790
lxdpool01/images/9dd39f26cb44777c9f577f4d4f87cc2de7c35cf6cf8e8acf7ac9172fd1087790@readonly 0B - 197M -
lxdpool01/snapshots 75.4M 98.3G 24K none
lxdpool01/snapshots/bitwarden 75.2M 98.3G 24K none
lxdpool01/snapshots/bitwarden/01-19-20-01-23 75.1M 98.3G 13.3G /var/snap/lxd/common/lxd/storage-pools/lxdpool01/containers-snapshots/bitwarden/01-19-20-01-23
lxdpool01/snapshots/docker-learn 24K 98.3G 24K none
lxdpool01/snapshots/docker01 24K 98.3G 24K none
lxdpool01/snapshots/ghost-02 24K 98.3G 24K none
lxdpool01/snapshots/ghost-dev 24K 98.3G 24K none
lxdpool01/snapshots/mx2 24K 98.3G 24K none
lxdpool01/snapshots/ns2 24K 98.3G 24K none
lxdpool01/snapshots/reverse-proxy 24K 98.3G 24K none
lxdpool01/snapshots/revproxy-restore 24K 98.3G 24K none
lxdpool01/snapshots/revproxy02 24K 98.3G 24K none
lxdpool01/snapshots/unms 24K 98.3G 24K none
lxdpool01/tmp 24K 98.3G 24K /lxdpool01/tmp
lxdpool01/virtual-machines 24K 98.3G 24K none
And some more debug log of an container
lxc revproxy02 20201005105631.764 TRACE commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket
lxc revproxy02 20201005105631.765 TRACE start - start.c:lxc_init_handler:694 - Created anonymous pair {3,7} of unix sockets
lxc revproxy02 20201005105631.765 TRACE commands - commands.c:lxc_cmd_init:1671 - Created abstract unix socket "/var/snap/lxd/common/lxd/containers/revproxy02/command"
lxc revproxy02 20201005105631.765 TRACE start - start.c:lxc_init_handler:709 - Unix domain socket 9 for command server is ready
lxc revproxy02 20201005105631.766 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:969 - Set process title to [lxc monitor] /var/snap/lxd/common/lxd/containers revproxy02
lxc revproxy02 20201005105631.767 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 4
lxc revproxy02 20201005105631.767 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 5
lxc revproxy02 20201005105631.767 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 6
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_start:2113 - Doing lxc_start
lxc revproxy02 20201005105631.767 INFO lsm - lsm/lsm.c:lsm_init:30 - LSM security driver AppArmor
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_init:732 - Initialized LSM
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STARTING
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_init:738 - Set container state to "STARTING"
lxc revproxy02 20201005105631.767 TRACE start - start.c:lxc_init:794 - Set environment variables
lxc revproxy02 20201005105631.767 INFO conf - conf.c:run_script_argv:343 - Executing script "/proc/30271/exe callhook /var/snap/lxd/common/lxd 47 start" for container "revproxy02"
lxc revproxy02 20201005105631.767 TRACE conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc revproxy02 20201005105631.767 TRACE conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc revproxy02 20201005105631.768 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:830 - First child 30734 exited
lxc revproxy02 20201005105631.884 DEBUG conf - conf.c:run_buffer:313 - Script exec /proc/30271/exe callhook /var/snap/lxd/common/lxd 47 start produced output: Error: Get "http://unix.socket/internal/containers/47/onstart": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
lxc revproxy02 20201005105631.886 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc revproxy02 20201005105631.886 ERROR start - start.c:lxc_init:798 - Failed to run lxc.hook.pre-start for container "revproxy02"
lxc revproxy02 20201005105631.886 ERROR start - start.c:__lxc_start:1945 - Failed to initialize container "revproxy02"
lxc revproxy02 20201005105631.886 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to ABORTING
lxc revproxy02 20201005105631.886 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc revproxy02 20201005105631.886 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STOPPING
lxc revproxy02 20201005105631.886 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc revproxy02 20201005105631.886 INFO conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 47 stopns" for container "revproxy02"
lxc revproxy02 20201005105631.887 TRACE conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=stop
lxc revproxy02 20201005105631.887 TRACE conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc revproxy02 20201005105631.966 DEBUG conf - conf.c:run_buffer:313 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 47 stopns produced output: Error: Get "http://unix.socket/internal/containers/47/onstopns?target=stop&netns=": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
lxc revproxy02 20201005105631.969 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc revproxy02 20201005105631.969 ERROR start - start.c:lxc_end:923 - Failed to run "lxc.hook.stop" hook
lxc revproxy02 20201005105631.969 TRACE start - start.c:lxc_end:943 - Closed command socket
lxc revproxy02 20201005105631.969 TRACE start - start.c:lxc_end:954 - Set container state to "STOPPED"
lxc revproxy02 20201005105631.969 INFO conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/lxcfs/lxc.reboot.hook" for container "revproxy02"
lxc revproxy02 20201005105631.969 TRACE conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc revproxy02 20201005105631.969 TRACE conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc revproxy02 20201005105632.475 INFO conf - conf.c:run_script_argv:343 - Executing script "/snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 47 stop" for container "revproxy02"
lxc revproxy02 20201005105632.475 TRACE conf - conf.c:run_script_argv:386 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc revproxy02 20201005105632.475 TRACE conf - conf.c:run_script_argv:391 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc revproxy02 20201005105632.632 DEBUG conf - conf.c:run_buffer:313 - Script exec /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd 47 stop produced output: Error: Get "http://unix.socket/internal/containers/47/onstop?target=stop": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
lxc revproxy02 20201005105632.635 ERROR conf - conf.c:run_buffer:324 - Script exited with status 1
lxc revproxy02 20201005105632.635 ERROR start - start.c:lxc_end:965 - Failed to run lxc.hook.post-stop for container "revproxy02"
lxc revproxy02 20201005105632.635 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:841 - No such file or directory - Failed to receive the container state
lxc 20201005105632.654 TRACE commands - commands.c:lxc_cmd:290 - Connection refused - Command "get_state" failed to connect command socket
lxc revproxy02 20201005105631.884 DEBUG conf - conf.c:run_buffer:313 - Script exec /proc/30271/exe callhook /var/snap/lxd/common/lxd 47 start produced output: Error: Get "http://unix.socket/internal/containers/47/onstart": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
That's very very unusual. Are you using some kind of bind-mount for data under /var/snap?
stgraber
on 5 Oct 2020
@stgraber No, no bind mounts here.
dkruyt
on 5 Oct 2020
Ok. If that's an acceptable solution for you, I'm 99% sure that a reboot would do it.
If restarting the system isn't an option, I'm sure we can find some weird ways to get the socket to behave inside the snap environment but that will need some more poking around to see how it broke.
stgraber
on 5 Oct 2020
Rebooting in 3.2.1...
Yes, lxd is working again, only needed to manually start the containers, no containers where started automatically.
Thanks...
dkruyt
on 5 Oct 2020
You may want to stash boot.autostart=true in your default profile if you want to ensure all containers are always started, regardless of their last state.
stgraber
on 5 Oct 2020
Related issues
jsnjack
·
3Comments
srkunze
·
3Comments
purell
·
4Comments
iteco
·
3Comments
rrva
·
5Comments